📚    BPM    SDLC    BIAanl    Data    Meta    Math    📚 👐 🎭 index - references    elucidation    metier 🎭
⚒    Intro    data meaning    structuring    inbound dwh    outbound dwh    What next    ⚒ 👐    top bottom   👐

Design Data - Information flow


information, data: enterprise core objects.

Data, gathering information on processes.

BI life Value stream data Data informa types The data explosion. The change is the ammount we are collecting measuring processes as new information (edge).

📚 Information questions.
⚙ measurements data figures.
🎭 What to do with new data?
⚖ legally & ethical acceptable?

🔰 Most logical back reference.

Contents

Reference Topic Squad
Intro Data, gathering information on processes. 01.01
data meaning The meaning of data. 02.01
structuring Structuring information or structuring data. 03.01
inbound dwh Edwh 3.0 - Data: collect - store. 04.01
outbound dwh Edwh 3.0 - Data: - store - deliver. 05.01
What next Change data - Transformations. 06.00
Combined pages as single topic. 06.02

Combined links
Combined pages as single topic.
🕶 info types different types of information
info types different types of data
👓 Value Stream of the data as product
👓 transform information data inventory
👓 data silo - BI analytics, reporting

🔰 Most logical back reference.

Progress


dual feeling

The meaning of data.


Everybody is using a different contact to the word "data". That is confusing when trying to do something with data.
As the datacentre is not a core business activity for most organisations there is move in outsourcing (cloud).


df_machines.jpg
Data Technical - machines oriënted.
The technical machines oriënted approach is about machines and the connections between them (network).

The service of delivering Infrastructure (IAAS) is limited to this kind of objects. Not how they are inter related.

The problem to solve behind this are questions of:


etl-reality.jpg
Information process oriënted.
The information process has many interactions input and output flows. There is no relationsship to machines and networking. The problem to solve those interactions will popup at some point.
Issues by conversions in datatypes, validations in integrity when using segragated sources (machines) will popup at some point


Processes actions oriënted.
Focussing on the process with actions and their flow is very detailed on what is happening and whether that is what is wanted for the organisation.
ee-institue
The top of this figure is the process oriëntation being split is several more details
As anybody is using an ICT system somewhere the intention meaning and classification (CIA) of data can vary tremendously.

dual feeling

Structuring information or structuring data.


The technical solutions as first process option.
There is some technical addition to solving an information question. This is of all times because of not well evaluating the possible solutions for the questions. Sometimes a simple paper note will do, sometimes an advanced new machine is needed. It depends on the situation. A simple solution avoiding the waste is lean - agile

archive documents nosql
Optimization Transactional Data.
The way of processing information was in the old day using flat files in the physical way. Still very structured stored and labelled.

In the modern approach these techniques still are applicable. The "NO SQL" hype is a revival of choosing more applicable techniques.
It is avoiding the transactional RDBMS approach as the single possible technical solution.

olap_star01.jpg
Optimization OLTP processes.
The relational SQL DBMS replaced codasyl network databases (see math).

The goal is simplification of online transaction processing (oltp) data by deduplication and normalization (techtarget) using DBMS systems supporting ACID ACID properties of transactions (IBM).

These approaches are necessary doing database updates with transactional systems. Using this type of DBMS for analytics (read-only) was not the intention.
normalization (techtarget, Margaret Rouse ) Database normalization is the process of organizing data into tables in such a way that the results of using the database are always unambiguous and as intended. Such normalization is intrinsic to relational database theory. It may have the effect of duplicating data within the database and often results in the creation of additional tables.
ACID properties of transactions (IBM)
DBMS changing types
A mix of several DBMS are allowed in a EDWH 3.0. The speed of transport and retentionperiods are important considerations. Technical engineering for details and limitations to state of art and cost factors.
dbmsstems_types01.png
dual feeling

Edwh 3.0 - Data: collect - store.


The DWH 3.0 as core business factor.
In the real world companies exist that are relying on the sole of just moving stuff very efficient (Amazon, Maersk).
Understanding those efficient approaches gives the opportunity to learn from them. Clean lean manufacturing (TPS) is the other ingredient.

allaboutlean: Line Layout Strategies  Part 1: The Big Picture Goods are coming in and goods are to be delivered.

A data warehouse should be central of any information system.

Very basic and appropriate to any ICT system. The time scale and implementation may vary on requirements by the special case. Inbound and outbound can be separated, responsibilities can get separated.

allaboutlean: Line Layout Strategies  Part 1: The Big Picture Having a single warehouse is also possible.
The transportlines is the manaufacturing process.
Manufactoring is the same thing as the information process when that is the real value line.
The transports are different at lean optimization. Inventory transport supermarkets during manufactoring.
Goal: Reducing transport waste.

Focus on the collect - receive side.
There are many different options how to receive iformation, data processing. Multiple sources of data - Multiple types of information.

Designing edwh 3.0 in a picture:
df_collect01.jpg
allaboutlean: Kanban Card Design  Material Flow-Related Information
Transport in manufactoring.
Processing objects, information data is usually crossing amany applications systems. This involves data transaport data conversions and will require data validation.

Any kind of transport that can be avoided of collected to be minimal will result in a lean process.

Mixing up the transport with a manufactoring step doesn't make sense. Using ETL (extract Transform Load) that manufacturing is introduced. Only needed (packaging) conversions are allowed.
The duality is that a disruptive break is needed in the technical machine approach to achieve this.

dual feeling

Edwh 3.0 - Data: - store - deliver .


What is delivered in a information process?
For a long time the only delivery of an information process was a hard copy paper result.
printing delivery line The mailing print processing is the oldest Front-end system using Back-end data. The moment of printing not being the same of the manufactured information.

Many more frontend deliveries have been created recent years. The domiant ones becoming webpages and apps on smartphones.
A change in attitude is needed bu still seeing it as a delivery needed the quality of infomration by the process.


Focus on the ready - deliver side.
There are possible many data consumers.
It is all about "operational" production data".
A classification by consumption type:

Operations For goals where standard systems are not appropiate or acting as an interface for not coupled systems. 💰
Results are input for other data consumers. Sensitive data allowed (PIA).
Archive of data - information not anymore available in operations, only for limited goals and associated with a retention period. ⚖
Business Intelligence (reporting). Developping and generating reports for decsion makers. Possible ias usage of analytical tools with DNF. ✅
Sensitive data is eliminated as much is possible. <
Analytics Developing Machine Learning. ❗ This is: ALC type3.
Sensitive data is eliminated as much is possible.
Analytics, Operations Machine Learning. ❗ This is: ALC type3.
Sensitive data may be used controlled (PIA).
Results are input for other data consumers.

Designing EDWH 3.0 in a picture:
df_delivery01.jpg

 horse sense

Change data - Transformations


ETL ELT - No Transformation.
etl-elt_01.png Transforming data should be avoided, it is the data-consumer process that should do logic processing.
The era offlaoding data, doing the logic in Cobol before loading, historical.

Servicing data delivery at different speed.
Diagram_of_Lambda_Architecture_generic_.jpg All kind of data (technical) should get support for all types of information (logical) at all kinds of speed.
Speed, streaming, is bypassing (duplications allowed) the store - batch for involved objects. Fast delivery (JIT Just In Time). lambda architecture. (wikipedia).


Combined pages as single topic.
Combined links
🕶 info types different types of information
info types different types of data
👓 Value Stream of the data as product
👓 transform information data inventory
👓 data silo - BI analytics, reporting


🔰 Most logical back reference.



⚒    Intro    data meaning    structuring    inbound dwh    outbound dwh    What next    ⚒ 👐    top bottom   👐
📚    BPM    SDLC    BIAanl    Data    Meta    Math    📚 👐 🎭 index - references    elucidation    metier 🎭

© 2012,2019 J.A.Karman