Design Data - Information flow

🎭 index references    elucidation    metier 🎭
👐 top    mid    bottom   👐

📚   BPM   SDLC   BIAanl   Data   Meta   Math   📚
  
⚖   Intro   data meaning   structuring   inbound dwh   outbound dwh   What next   ⚖

information, data: enterprise core objects.

Data, gathering information on processes.

BI life Value stream data Data informa types The data explosion. The change is the ammount we are collecting measuring processes as new information (edge).

📚 Information questions.
⚙ measurements data figures.
🎭 What to do with new data?
⚖ legally & ethical acceptable?

🔰 Most logical back reference.

Contents

Reference Topic Squad
Intro Data, gathering information on processes. 01.01
data meaning Enterprise engineering, valuable processing flows. 02.01
structuring Information - data - avoiding process fluctuations. 03.01
inbound dwh Edwh 3.0 - Data: collect - store - deliver. 04.01
outbound dwh Patterns by changing context, changing technology. 05.01
What next Change data - Transformations. 06.00
Combined pages as single topic. 06.02

Combined links
Combined pages as single topic.
🕶 info types different types of information
🚧 info types different types of data
👓 Value Stream of the data as product
👓 transform information data inventory
👓 data silo - BI analytics, reporting

Progress


d´Agapeyeff&Acute Inverted Pyramid.
dual feeling

Enterprise engineering, valuable processing flows.

Everybody is using a different contact to the word "data". That is confusing when trying to do something with data. A mind switch is seeing it as information processing in enterprises. As the datacentre is not a core business activity for most organisations there is move in outsourcing (cloud SAAS).
 
valuable processing flow.
Call it demand/supply, Customer/Producer, requestor deliverer, there is a circle in the process.
transaction Dietz 2016 handout presentation.
 
The product delivery is right to left, that is confusing.
The producer is the one that is an contact with the customer / requestor. The front-end in an organisation. What is all behind (backend) is not made visible.
ee-institue An organisation has four phases in activities when the product is produced. Product flow is left to right, the cyclus clockwise.
 
The customer is coming and leaving at the bottom right side. He is only seeing the front end customer desk.
High level strategy management is in the middle, control at the bottom, at the top the transformation / manufacturing .
Processes, transactions oriënted.
ee-institue 2015-07-17 Dietz, Jan L.G. - Concise Summary of DEMO-3
Focussing on the process with actions and their flow. This is very detailed on what is happening and whether that is what is required for the organisation by decompositions.

etl-reality.jpg Just reversing the shape of triangle keeping the the colours similar.
Assuming the enterprise transactions to produce are all in the now biggest red part of the triangle.
The other colours - layers, have several options (multiple dimensions). The option for using layered coordination is the one that is a fit when a management information system is the question.

demo ducplicated to 4 sides Replacing the noisy details in the previous figure with four phases in a circle results in a more abstract version.
The internal product flow and organisational silos are clearly visible with a lot of areas where gaps and miscommunications are to be found.
 
What is not visble is the external productflow and how to solve fluctuations in customer demand. The Three Fundamental Ways to Decouple Fluctuations inventory, capacity, and time. Each of them has different advantages and disadvantages. 🔰 Inventory is a main first and last point in a flow.
dual feeling

Information - data - avoiding process fluctuations.

Engineering a process flow, then at a lot of point there will be waits. At the starting and ending point it goes from internal to external where far longer waits to get artefacts or product deliveries will happen. Avoiding fluctuations having a predictable balanced workload is the practical solution to become effciënt.
💣 The role of the EDW 3.0 is enterprise operationals value stream. It is not something being reserved for reporting purposes (BI AI).
 
Incoming - Manufacturing backend - Outgoing front end.
allaboutlean: Line Layout Strategies ? Part 1: The Big Picture Every process has a moment of the request coming in. Something is to get delivered as response. The input en output definitions of the work transformation be get done. A data warehouse should be central of any information system, the operational system, value stream flow.

allaboutlean: Line Layout Strategies ? Part 1: The Big Picture The incoming and outgoing information is possible a shared one. That will require: there is not a big difference in artifact types the frontend of delivery and receiving requests.
The manufacturing (backend) is the organisation working floor where the real processing is done.
The line lay out for a manufacturing process is having a clear starting and ending point. These edge point are named "warehouses". Lean work is not about eliminating the warehouse or not having any inventory. It is about minimizing the waste.

THe EDWH 3.0 Logistics as basic central pattern.
Having a inbound area the validation of goods, infomation, is done.
At the manufacturing side are the internal organisation consumers. Not only for a dashboard to be used by managers but all kind of consumers including operational lines.
df_csd01.jpg
The two vertical lines are managing whos has acces to what kind of data, autorized by dataowner, registered data consumers, monitored and controlled.
The confidentiality and integrity steps are not bypassed with JIT (lambda).

Inventory one of the options - Levelling (Heijunka)
allaboutlean: The Three Fundamental Ways to Decouple Fluctuations The value stream with all transformations and interactions is not that predictable. Managing fluctuations is important. Why Levelling (Heijunka) is important
Using inventory is probably the easiest way to have a structured decoupling of fluctuations. You can add it pretty much between every process to buffer the fluctuations between the processes.
Another way to decouple fluctuation is by adjusting your capacity. - The problem with adjusting capacity is the delay between the decision to increase or reduce capacity, and the actual increased or reduced capacity.
💰 If you didn’t manage to decouple using buffer or capacity, eventually somebody has to wait. This may be either the customer (if demand is larger than capacity) or your operators and machines (if demand is less than capacity).
Structure for Reducing Fluctuations
Government Organisation using technology.
This has nothing to do with hard facts but everything with things like my turf and your fault. Parties have their preferences on availability confidentiality integrity. Using shared technology easily runs into conflicts.
🤔 The most understandable question as result is having a dedicated machine for each "application". This is not sensible when the information process as core business gets broken by disparate connections between machines.
🤔 Clear decoupling needed: machines services and "applications".

dual feeling

Edwh 3.0 - Data: collect - store - deliver.

Processing objects, collecting information and delivering goes along with responsibilities. It is not sexy, infact rather boring. Without good implementation all other activities are easily getting worthless. The biggest successed like Amazon are probably more based in doing this very well than something else. The Inner Workings of Amazon Fulfillment Centers
 
Focus on the collect - receive side.
There are many different options how to receive information, data processing. Multiple sources of data - Multiple types of information.
df_collect01.jpg In a picture:
 
A data warehouse should be the decoupling point of incoming and outgoing information.
 
A data warehouse should validate verify the delivery on what is promised to be there. Just the promise according to the registration by administration, not the quality of the content (different responsibility).

allaboutlean: Kanban Card Design ? Material Flow-Related Information
Transport in manufacturing.
Processing objects, information data is usually crossing many applications systems. This involves data transport data conversions and will require data validation.
 
Any kind of transport that can be avoided of collected to be minimal will result in a lean process.
 
Mixing up the transport with a manufacturing step doesn´t make sense. Using ETL (extract Transform Load) that manufacturing is introduced. Only needed (packaging) conversions are allowed.
The duality is that a disruptive break is needed in the technical machine approach to achieve this.

Focus on the ready - deliver side.
A classification by consumption type:
df_delivery01.jpg In a picture:
 
There are possible many data consumers.
It is all about "operational" production data" - production information.
 
Some business applications only are possible using the production information.

dual feeling

Patterns by changing context, changing technology.

Common used ICT patterns processing information. For a long time the only delivery of an information process was a hard copy paper result. Deliveries of results has changed to many options. The storing of information has changed also.
 
The technical solutions as first process option.
Sometimes a simple paper note will do, sometimes an advanced new machine is needed. It depends on the situation. A simple solution avoiding the waste is lean - agile
archive documents nosql Optimization Transactional Data. An warehouse does not content structuring it must be able to locate the wanted content structured. Delivering the labelled containers efficient >
Optimization Transactional Data. The way of processing information was in the old day using flat files in the physical way. Still very structured stored and labelled. In the modern approach these techniques still are applicable although automated hidden in a RDBMS .
Analytics & reporting. The "NO SQL" hype is a revival of choosing more applicable techniques.
It is avoiding the transactional RDBMS approach as the single possible technical solution.

etl-reality.jpg
Information process oriënted, Process flow.
The information process in an internal flow has many interactions input, transformations and output in flows.
There is no relationship to machines and networking. The problem to solve those interactions will popup at some point.
Issues by conversions in datatypes, validations in integrity when using segregated sources (machines) will popup at some point.

The service bus (SOA).
SD_enterpriseservicebus.jpg ESB enterprise service bus The technical connection for business applications is preferable done by a an enterprise service bus. The goal is normalized systems.
Changing replacing one system should not have any impact on others.

Microservice_Architecture.png
Microservices with api´s
Microservices (Chris Richardson):
Microservices - also known as the microservice architecture - is an architectural style that structures an application as a collection of services that are: The microservice architecture enables the continuous delivery/deployment of large, complex applications. It also enables an organization to evolve its technology stack.

Data in containers.
informatie_mdl_imkad11.jpg Data modelling using the relational or network concepts is based on basic elements (artefacts).
An information model can use more complex objects as artefacts. In the figure every object type has got different colours.
The information block is a single message describing complete states before and after a mutation of an object. The Life Cycle of a data object as new metainformation. Any artefact in the message following that metadata information.
This is making a way to process a chained block of information. It is not following the blockchain axioma´s. The real advantage of a chain of related information is detecting inter-relationships with the possible not logical or unintended effects.

olap_star01.jpg
Optimization OLTP processes.
The relational SQL DBMS replaced codasyl network databases (see math). The goal is simplification of online transaction processing (oltp) data by deduplication and normalization (techtarget) using DBMS systems supporting ACID ACID properties of transactions (IBM).
These approaches are necessary doing database updates with transactional systems. Using this type of DBMS for analytics (read-only) was not the intention.
normalization (techtarget, Margaret Rouse ) Database normalization is the process of organizing data into tables in such a way that the results of using the database are always unambiguous and as intended. Such normalization is intrinsic to relational database theory. It may have the effect of duplicating data within the database and often results in the creation of additional tables.
ACID properties of transactions (IBM)
 horse sense

Change data - Transformations

Working on a holistic approach on information processing starting at the core activities can solve al lot of problems. Why just working on symptoms and not on root causes?
💡 Preparing data for BI, Analytics has become getting an unnecessary prerequisite. Build a big design up front: the enterprise data ware house (EDWH 3.0).
 
Data Technical - machines oriënted.
The technical machines oriënted approach is about machines and the connections between them (network). The service of delivering Infrastructure (IAAS) is limited to this kind of objects. Not how they are inter related.
The problem to solve behind this are questions of:

df_machines.jpg 🤔 A bigger organisations has several departments. Expectations are that their work has interactions and there are some central parts.
Sales, Marketing, Production lines, bookkeeping, payments, accountancy.
🤔 Interactions with actions between all those departments are leading to complexity.
🤔 The number of machines and the differnces in stacks are growing fast. No matter where these logical machines are.
For every business service an own dedicated number of machines will increase complexity.

The information process flow has many interactions, inputs, tranformtions and outputs.
💡 Reinvention of a pattern. The physical logistic warehouse approach is well developed and working well. Why not copy that pattern to ICT? (EDWH 3.0)

printing delivery line
What is delivered in a information process?
The mailing print processing is the oldest Front-end system using Back-end data. The moment of printing not being the same of the manufactured information.

Many more frontend deliveries have been created recent years. The domiant ones becoming webpages and apps on smartphones.
A change in attitude is needed bu still seeing it as a delivery needed the quality of infomration by the process.

Government Organisation using technology.
This has nothing to do with hard facts but everything with things like my turf and your fault. Different responsible parties have their preferences on availability confidentiality integrity. Using shared technology is easily getting into conflicts.
🤔 The most undstandable for having a dedicated machine. Not sensible when the information process as core business gets broken.
🤔 Clear decoupling in responsibility is needed (SAAS).

ontological PDCA cycle
PDCA cycle redefined as ontology
In the business development area a full SDLC life cycle is described with many topics with three power lines of involved parties. In every area of the three colored segments a representation of the basic power lines is found.
The SDLC process life cycle is based on four quadrants, not three segments. The is not an easy fit tot align. Two addition small representatives are needed in the upper quadrants (I,II) whereas the lower (III,IV) it is already shared.
It is far more difficult to adjust an existing process than just building a new one. The horizontal compliancy lines (internal exterenal) are possible adjusting what is in progrees. The vertical alignment lines are possible adjusting what is in progrees.
Combined pages as single topic.
Combined links
🕶 info types different types of information
info types different types of data
👓 Value Stream of the data as product
👓 transform information data inventory
👓 data silo - BI analytics, reporting


🔰 Most logical back reference.


⚖   Intro   data meaning   structuring   inbound dwh   outbound dwh   What next   ⚖
  
📚   BPM   SDLC   BIAanl   Data   Meta   Math   📚

© 2012,2020 J.A.Karman
👐 top    mid-1    split    mid-2    bottom   👐
🎭 index references    elucidation    metier 🎭