📚    BPM    SDLC    BIAanl    Data    Meta    Math    📚 👐 🎭 index - references    elucidation    metier 🎭
⚒    Intro    logistics    patterns    performance    provision    What next    ⚒ 👐    top bottom   👐

Design Data - Information flow


information, data: enterprise core objects.

Data, central points of information with processes.

BI life workcell Data informa types The data provision with the data explosion is needing a strcutured approach in a value stream.

📚 Information questions.
⚙ measurements data figures.
🎭 What to do with new data?
⚖ legally & ethical acceptable?

🔰 Most logical back reference.

Contents

Reference Topic Squad
Intro Data, central points of information with processes. 01.01
logistics Logistics of the EDWH 3.0 02.01
patterns Common used ICT patterns processing information. 03.01
performance EDW performance challenges. 04.01
provision Delivery Information at the manufacturing floor. 05.01
What next Change data - Transformations. 06.00
Combined pages as single topic. 06.02

Combined links
Combined pages as single topic.
👓 info types different types of data
👓 Value Stream of the data as product
🚧 transform information data inventory
👓 data silo - BI analytics, reporting

Progress


 legal

Logistics of the EDWH 3.0

Processing objects, information goes along with responsibilities.
A data warehouse should be the decoupling point of incoming and outgoing information.
A data warehouse should validate verify the delivery on what is promised to be there. Just the promise according to the promise according administration, not the quality of the content (different responsibility).

Incoming front end - Manufacturing backend - Outgoing front end.
allaboutlean: Line Layout Strategies – Part 1: The Big Picture Goods are coming in and goods are to be delivered.

A data warehouse should be central of any information system.

allaboutlean: Line Layout Strategies – Part 1: The Big Picture The incoming and outgoing information is possible a shared one. That will require there is not big difference in the frontend of delivery and receiving requests.

The manufacturing (backend) is the organisation working floor where the real processing is done.


THe EDWH 3.0 Logistics as basic central pattern.
Having a inbound area the validation of goods, infomation, is done.
At the manufacturing side are the internal organisation consumers. Not only for a dashboard to be used by managers but all kind of consumers including operational lines.
df_csd01.jpg
The two vertical lines are managing whos has acces to what kind of data, autorized by dataowner, registered data consumers, monitored and controlled.
The confidentiality and integrity steps are not bypassed with JIT (lambda).


dual feeling

Common used ICT patterns processing information.

There are a lot of technical approaches to exchange information within ICT.
All those technical solutions are mostly limited to an dedicated external supplier within an intended "application" use.

SD_enterpriseservicebus.jpg
The service bus (SOA).
ESB enterprise service bus The technical connection for business applications is preferable done by a an enterprise service bus. The goal is normalized systems.
Changing replacing one system should not have any impact on others.

Microservice_Architecture.png
Microservices with api´s
Microservices (Chris Richardson):
Microservices - also known as the microservice architecture - is an architectural style that structures an application as a collection of services that are: The microservice architecture enables the continuous delivery/deployment of large, complex applications. It also enables an organization to evolve its technology stack.

informatie_mdl_imkad11.jpg
Data in containers.
Data modelling using the relational or network concepts is based on basic elements (artifacts).

An information model can use more complex objects as artifacts. In the figure every object type has got different colors.

The information block is a single message describing complete states before and after a mutation of an object. The Life Cycle of a dataobject as new metainformation. Any artifact in the message following that metadatainformation.
This is making a way to proces a chained block of information. It is not following the blockchain axioma´s. The real advantage of a chain of related information is detecting inter-relationships with the possible not logical or unintended effects.

etl-elt_01.png
ETL ELT - EL

Classic is the processing order:
⌛ Extract, ⌛ Transform, ⌛ Load.

For segregation a technical copy is commonly required.
Issues are:

Combined links
Needed building blocks in other design lines.
👓 DTAP Multiple dimensions processes by layers
👓 ALC type 3 low code ML process development
👓 data administration describing modelling data
👓 Security - modelling access information


 legal

EDW performance challenges.


Tuning performance basics.
Solving performance problems requires understanding of the operating system and hardware. That architecture was set by von Neumann (see design-math).
vonNeumann_perftun01.jpg
A single CPU, limited Internal Memory and the external storage.
The time differences between those resources are in magnitudes (factor 100-1000).

Optimizing is balancing between choosing the best algorithm and the effort to achieve that algorithm.

vonNeumann_perftun02.jpg
That concept didn´t change. The advance in hardware made it affordable to ignore the knowledge of tuning.

The Free Lunch Is Over .
A Fundamental Turn Toward Concurrency in Software, By Herb Sutter.
If you haven’t done so already, now is the time to take a hard look at the design of your application, determine what operations are CPU-sensitive now or are likely to become so soon, and identify how those places could benefit from concurrency. Now is also the time for you and your team to grok concurrent programming’s requirements, pitfalls, styles, and idioms.

Additional component, the connection from machine, multiple cpu´s - several banks internal memory, to multiple external storage boxes by a network.

Perftun_EtL01.jpg
Tuning cpu - internal memory.
Minimize resource usage: ❗ The "balance line" algorithm is the best. A DBMS will do that when possible.

Perftun_EtL02.jpg
Network throughput.
Minimize delays, use parallelization:
⚠ Transport buffer size is a coöperation between remote server and local driver. The local optimal buffer size can be different. Resizing data in buffers a cause of performance problems.

Perftun_EtL03.jpg
Minize delays in the storage system.
⚠ Using Analtyics, tuning IO is quite different to transactional DBMS usage.
💣 This different non standard approach must be in scope with service management. The goal of sizing capacity is better understood than Striping for IO perfromance.


 
dual feeling

Delivery Information at the manufacturing floor.

Having many type of consumers the service for delivery ant to who deliver is next.

df_dlv_bi-anl
BI reporting delivery

Building up reports is / was the only consumer for a dwh. The transformations and data modelling to be able to report are complex. These approaches are resulting in expensive and difficult to manage systems.

The manufacturing process is sometimes needing information (historical records) that as limitations on the operational systems were never realised in those systems.
As a result for solving those operational questions connections were made

Archiving historical records that may be retrieved is an option that should be a regular consumer.


df_dlv_alctp3
Analytics ML - operations delivery

The analytics AI ML machine learning has an duality.
The modelling stage (develop) is using data, that data is not the same, although similar, as in the operational stage.
Developing is done with operational production data. The sizing van be much bigger than that of what is needed at operations due to the needed history.

The results of what an operational model is generating should be well monitored for many reasons. That is new information to process.

The way of developping using ML is exchanging some roles in coding and data to achieve results
(ALC type3)


wrh_selfsrvc-01.jpg
Selfservice - Managed

Self service sounds very friendly, it is a euphemism for no service. Collecting your data, processing your data, yourself.

Have it prepared transported for you so it can processed for you.
wrh_cntr_stor.jpg

 horse sense

Change data - Transformations


The use of a dwh can be hidden. During transport it is still inventory. Transport can take a long time to complete.

shp_cntr_clct.jpg
Logistics using containers.

= containers going to be transported =

Containers have become rapidly the standard in physical logistics.
It are not the objects being transported but containers.

shp_cntr_store.jpg
Time passing by during transport before delivery.
Planned calculations are needed for the service agreements.
shp_cntr_dlv.jpg
Almost ready for delivery.

= containers arrived, able to deliver from this point =


Combined pages as single topic.
Combined links
👓 info types different types of data
👓 Value Stream of the data as product
transform information data inventory
👓 data silo - BI analytics, reporting


🔰 Most logical back reference.



⚒    Intro    logistics    patterns    performance    provision    What next    ⚒ 👐    top bottom   👐
📚    BPM    SDLC    BIAanl    Data    Meta    Math    📚 👐 🎭 index - references    elucidation    metier 🎭

© 2012,2019 J.A.Karman