jakarman - ICT - My profession

Innovate & Operate

All organisations wants to be:

1. a game changer == new products, dropping old ones
2. being predictable and reliable. == same trustworthy product

The business usually wants it all at the same moment with no investment.
A conflict of interests that will last forever.

The choice:

• conservative running as-is (vertical)
  👉🏾 stable - operations
  👎🏾 no process changes, no innovation
• disruptive innovation (horizontal)
  👎🏾 instable operations
  👉🏾 process changes, innovations

Doing two major changes at the same time often results in unwanted surprises, scheduling actions for time is sensible.



..

I - Running Changing ICT

Having all years since 1980´s being active at ICT (Information Communication Technology). The focus is mainly "financial services".

Many things have changed. What has changed are the technical laysers. What has not changed is the expected service for the business by ICT.
The most important statement is keeping the lights on while everything is physical placed in rooms dat are preferable kept dark.

The fundamentals of information technology, mathematics, analytics hasn´t changed much.

• Code is being developed and maintained on business expectations.
• Data, information, is being processed and evaluated.

Achieving minor improvements even when required by compliancy rules isn´t that simple as of dependicies with impediments.


..

II - Experiences

I have mostly worked at the technical infrastructure (TI) in the ICT. Not siloed inside the ICT but connected to analytical usage.

Understanding, working with:

• Researchers, Analysts, Programmers at the business side
• Operating Systems, Tools, Security, Data, Databases, Datacommunications
• Setting up concepts and designs whith the ideas needed by business goals fullfilling compliancy issues knowing the technical limitations and possibilities
• Working in organizations with a more short term management goal. The needed long term visions are usually becoming challenges

A lot of the information questions and technical challenges have been kept always the same. The detailed levels are the hidden game changers. It looks like that what once was solved is forgotten now how to solve it.


..

III - Not working, Private

When I´m not at the office working, I could be off. When it is for a longer period that could be a holiday. Having my holidays, I´m charging the batteries.

A place where I can be is like this one.

I can get a telephone call, asking for help, from someone at the office.
After the support question handling off, the statement of my holiday at a nice location. With the modern smartphone tooling you will get a picture.
It has become an anecdotal story with old friends.

IV - Innovate, Closing the circle

The figure of the analaytics life cycle that is at the start of this paragraph, is a result of solving the issues having been confronted with.
It should cover the whole life cycle with all possible involved skills.

Developping a model (data science) is the what gets all the attention. The deployment evaluation and compliancy are not. To bring something through the whole cycle is requiring more capabilities than a single person is able and/or is allowed to do.

The common steps (design - devops):

• 📚 Information, == Ideas, associated information, Project
• 🎭 Communication == Architecting, Research, Explanation
• ⚙ Technology == Realization, Operations, Monitoring.

These three levels can be represented in a rectangle, better is a triangle

Roles

My profession (metier) is: ICT specialist . vcard: J A Karman
Roles are:

• Consultant
• System engineer - System programmer
• SAS platform administrator, engineer
• Many programming languages, including SAS

The working area is: support of customers(business) using analytics within the ICT (tactical - design) and to the details of the system engineering (technical - devops)

🎭 I won´t break my work at the point by just following detailed instructions what to do. I will listen for what I think is realy needed. After understanding the possibilities, going to solve that was is not asked explicitly, within the options that are available.

Contact

I am open for contacts, discussions

jakarman7w7         @ja_karman

You can contact me in case of some interesting questions. Social media names are shown above. E-mail mailto:info@jakarman.nl
I am living at Netherlands Gelderland - j.a.karman
..

Technical - Mathematics

My education is the elektro-technical engineering, added mathematics including statistics. Not quite a logical fit for information technology nowadays. At that moment courses for ICT didn't exist as it was building from scratch being a spinn-off with those mentioned technology backgrounds.

The microprocessor being used during one of the course lines was the 6502 (motorola) . My father had his connections with computing educating and mathematics. As the interest rates went up high for some years there was a problem with the than used calculation tabels became unusable. He did some circumventions using an Olivetti P101 for that at our home.

⚙ My first job was as "technical statistical support" role, helping researchers under supervision of staticians. I switched after some time

System programmer

My background has become the "system programmer".

The descriptions of roles by IBM (mainframe world): Mainframe sysprogrole System programmers are needed to install and maintain the middleware on the mainframe, such as database management systems, online transaction processing systems and Web servers.

Middleware is a software "layer" between the operating system and the end user or end user application.
It supplies major functions that are not provided by the operating system. Major middleware products such as DB2, CICS, and IMS can be as complex as the operating system itself, if not more so.

Changing words for same kind of work

Many positions are able to be found while using middleware, as it is technical, and needing for your business.
That nice IBM description mentioned IBM prodcuts. I was instead exposed to Cullinet (CA) IDMS DB/DC with an IDD (Integrated Data Dictionary) and in house made solutions. That IDD is a centralised metadata aproach using a network (not relational) DBMS and ER schema design.

The role of the system programmer has the same key points of the SAS platform administrator.
Someone has to understand the way to integrate those two worlds to get it effectively well working.

📚 In the Agile Scrum world, chapters guilds but I am missing the captain and business owner.

SAS - software product usage

My experience with SAS is going back as far as the early 1980.
This has become my major working area since 2004. The focus has been the way of implementing that and supporting SAS users (anlysts, marketing, data science) in operational environments.

My knowledge is kept up to date. With the certification program (2013 see personal notes). The order in getting those, was in line with my working approach, logically completely a reversed order.

Security - GDPR - Privacy

These topics are getting recently far more attention. They are a corner stone in my experiences. A mooc certification with the GDPR was done.

Chronological experiences

My working experiences are given in a chronological oreder for several typical periods. Interesting internal (external) links are made in the paragraphs.

Topics

I am using two viewpoints:

1. Design thinking.
   The technical details are eliminiated in a way you could exchange easily technologies.
   The Amsterdam Information model (AIM) is using tactical, strategy.
2. Devops details.
   The technical details how to realize and operate are in scope.
   The AIM model is using tactical, operational.

There are six defined topics and labeled "sense", that one for what makes sense but cannot given a sensable label.
To find your way I made a picture mindmap of that losely structure.

Mindmap Topics

..

Design & Devops references

The well ordered subjects in the mindmap, design & devops, have their own index and own starting point with a own glossary.

link	Design subject
bpm	Business Process Management organizing work
sdlc	Software Development Life Cycle Application Life Cycle ALM
bianl	Business Intelligence - descriptive Analytics - Predictive
sense	Common sense, what is not covered
data	Data storage, accessing data security
meta	Metadata & datalineage Data modeling
math	Mathematics, smart solutions standards policies

link	Devops subject
bpm	Business Process Management organizing work
sdlc	Software Development Life Cycle Application Life Cycle ALM
bianl	Business Intelligence - descriptive Analytics - Predictive
sense	Common sense, what is not covered
data	Data storage, accessing data security
meta	Metadata & datalineage Data modeling
math	Mathematics, smart solutions standards policies

..

Sharing knowledge

During my long time experiences some were very interesting as of concept and implementations. Experiences are to be presented by the periods with a situations and actions.

The references are part of the well ordered subjects but the reference looks to originate from a neverland location.

What I am sharing as personal knowledge and personal experiences may be shared, not freely copied without reference.

2019 march:

• Im am redesigning site and content.
• There could be something get wrong and old content being lost.
• The still relevant old content I want to convert, but that will need a lot of time. New content like the analytics life cycle (machine learning), has queued last years, that is my priority.

Old reference: My old 2013 site being to converted



..

The wild bunch - Design

link	Detailed description
ml_opr	Process engineering - operations research - Machine learning
ml_lcm	Analytics Life Cycle - Machine learning

..

The wild bunch - Devops

link	Detailed description
JST	(jaap´s) Job Submit Tool

Solving questions & challenges where no solution is in place is going where nobody has been before.

A limited number will be offered with an immediate link, reference from this page.

IVO (ISS, Individual Sales Support)

📚 The business goal was delivering pc´s to the sales people (600) at their home having all the information they needed for their customers (1984).
🎭 The avaibable technology was:

• pc´s were just coming into the market. Specificiations: processor 8086, memory 640Kb ram, harddisk 10Mb, OS Pc/Dos 2.1.
• Connected to an IBM mainframe, screenscaping Irma 3270 cards.
• A modified Viditel modem 1200/75 bps for calling out to the pc´s at home. Planned schedule time evening hours. limit duration 4 hours.
• Controlling environment operators task within the IDMS DB/DC (database backend and fronted).
• Using languages: batch script, Assembler, C, Cobol, ADS (IDMS).

⚙ What was done:

• Building Implementing and supporting the protocol for the data exchange between pc's. Optimized within technical limitations switching directions being 0,2s.
• A b-tree (no_sql) like dbms was build in C for doing the DBMS fucntionality. Extended maintenance was done in indexes and data blocks, improving performance & reliablity.
• Connecting PC-world to Mainframe with dedicted calling out pc´s.
• Supporting developers testers, a dedicated helpdesk and edcuational staff for what popped up.

Performance & Tuning, Mainframe

📚 The goal was setting up, delivering Management Infomation (MIS EIS) by applications and tools in system resource usage.
Multiple goals:

• Pinpointing on possible issues that could be improved, delaying exteral investments
• Able to plan, just in time, costly hardware upgrades
• Cost distribution to the several Business lines based on measured usage. (life -, property insurance)

🎭 The avaibable technology was:

• Needing to aqcuire tools. Chosen was SAS (5.17) with MXG to process the SMF records.
• Configuring adjusting settings for the SMF record processing.
• In house available scheduling and system resources.

⚙ Having done:

• Building up a weekly PDB (Performance Database) in several incrementals runs with a weekly archive and cleanup. All detailed weekly information going into this dataset.
• Appending all summaries into a SDDB (System Data Database) the cost account being added and minimized and summarized.

JST, a Generic approach for automated testing

📚 It is automating manual work of several IT-staff lines.
This is a very unusual part of IT optimization as it is internal IT. This is a solution I implemented around 1996 and is was still used in 2011 (unchanged).
Still after all this years is considered to be a very modern approach.
⚖ Experiences:

• With this approach the generating of the statements and the re-usage in development & testing is possible to a highly optimized process.
• It survived all years several reorganizations, the year 2000 testing, Euro testing, outsourcing to India.
• Not everybody was accepting the automatization of ICT work. Not reaaly different to any high impact change to personal work.

🎭 Used technology and limitations:

• The available program language ISPF dialog with REXX running in a TSO environment was used.
• Because of the business optimization that is done, the whole is not available as a commercial solution rolling out at other environments.

⚙ The solution design and realization:

• see: 👓 JST for details
• Having a well-designed JCL (Job Control Language) in a DTAP (Develop Test Acceptance Production) approach.
• The components being setup fully in DTAP approach getting connected to release management, versioning for those processes and tools (in house build, Endevor).

Database IDD - IDMS DB/DC

This was using an advanced approach using Integrated Data Dictionaries (IDD's). The more modern word would be Metadata Databases doing some kind of Masterdata management. integrated backend (DB Database) and frontend (DC Datacommunication) tool middleware system. Being executed

These dictionaries had to be setup in a master (sys:11) and based from that master a logical DTAP segregated configuration (sys:66 sys:88 sys:77 sys:01).
Each of them having definitions:

• technical definitions like terminals.
• security eg: users groups and access control.
• business applications menu tailoring, business logic and data.

The supplier Cullinet was positioned as being prefered. In previous years the front end was not available from the supplier. An in house build front-end middelware system was still running (VVTS).

📚 Goal operational support (system programmer) in a small team setting.
🎭 Used technology: suppliers Cullinet later CA. and the IBM toolset.
⚙ What was done:

• Automating generating definitions, versioning releases.
• removing home build exits. testing combined with JST.
• Supporting devlopers and testers in achieving their goals.

Security (ACF2 RACF - MS AD )

Bringing Access to resources into roles, organizing who is allowed to what role and getting to is that person realy who is saying he is, is known as security. There is a master security administrator task (designing) and a restricted security amdinistrator task (execuring requests) for the required segregation in duties.

The old common approach is doing input validation before handing over to the tool by lack of integrated security. Roscoe was an multi user mainframe program editor. The roscoe adminstrator tried to implement security by parsing and than reject or allow the command. It never succeeded to became reliable (1985). In modern times we are parsing code (preventing code injection) trying to secure browser-usage (2020).

📚 Goals implementing security with tools:

• Manageable explainable centralized security.
• Monitoring with Incident Response. CSIRT

🎭 Used technology: suppliers "ADR" later "CA", IBM, Microsoft (AD).
⚙ What was done:

• Important in helping for the design of a logical acces setup aligned with the needed segregation by DTAP policies. (ACF2).
• Helped in the transition in the move from Roscoe to TSO, implementing ACF2. Major impact on te way of working by developers.
• Build an in house tool to support in a quick roll out on all involved platforms changing access rights (group role based).
• Did help in the transition from ACF2 (Supplier CA) to RACF (IBM).

Scheduling (UCC7 TWS - homebuild)

Scheduling Jobs in the old times was all manually hard work. Operators had to carry all the physical paper cards in time to the readers to execute them. Planning was done as prepartions on paper using previous experiences ont load and durations. The operator was the person you sould be friends with.

This all changed when that work became automated and shifted for some parts to persons inside the business (functional support, prodcution support).
There is big shift in time impacting jobs. Many kind of jobs have gone and replaced by others in te era of applying computers AI ML.

📚 Goals:

• Replacing manual activity by a robot program (scheduler).
• Reducing failed deadlines for job readiness by learning and predicting what is going on. Being aware in daily weekly monthly runs.

🎭 Used technology and limitations:

• UCC7 later CA7 was the CA scheduler, it was the first one being implemented.
• Tivoli (TWS ppc) is the replacement delivered by IBM replaced the CA7 software.
• There are many more options to trigger jobs. JST was home build just using standard tools.

⚙ The realization being adjusted in level of acceptance at departments.

Operational Risk (OpRisk),

This departments and the work on their support is interesting. OpRisk is doing things like Advanced Measurement Approach. Monte Carlo simulation modelling with public known situations is the way to go.
It requires releasemangement, full testing (including DR) and security policy alignment. At the moment of the delivery deadline (quarterly) it is a critical proces.

📚 Goals

• Installation & implementation accoridng to in house Risk management policies.
• Helping tot run those regular reliable and archiving old versions.
• Support developers in guidelines building & maintaining that special analyses prediction.

🎭 Used technology used and limitations:

• SAS 9.1.3 AIX, SAS 9.3 Linux. Licnesed under a Solution.
• Oracle connection started in first approach (removed later).

⚙ Having done:

• Being able to do most succesful for the business line.
• Several issues:
  1. Issue 1: The internal cost assesment became unexplainable high.
  2. Issue 2: Doing a DR test succesfull was almost impossible due to too many involved machines. The Oracle database loction.
  3. A strange quirk in dynamic prompting causing difficulties to run for different lines (tenants) having their own internal data.

Hosting, multi tenancy - *AAS - stacks

Software As A Service (SAAS) is a great idea. Implement it by yourself if your requirements are more strictly then can be fulfilled with ´SAS on demand eg cloud services. Dot it by yourself when you are you are big enough to do a SAAS implementation yourself. Another reason can be having multiple buisness lines (tenants) needign tehe same solution teh same application.

Sharing computer resources can bring huge benefits. There is a whole industry based on that. When doing it in the wrong way the possible risks are also high.
Solving the SAS environment challenges with all my knowledge and experience it is brought to a much higher level as is common practice.

📚 Goals

• Definining borders of segreagtions in logical security.
• Definining release management for used tools & middelware.
• Definining release management for business applications.
• Segregation of duties in support and adminitration for ning relases management for used tools & middelware.

🎭 Used technology doesn´t really matter, it is about dependicies in the stack.
⚙ Having implemented:

• Achieved hosting (*AAS) with SAS supporting a full services.
• Got alignment with security policies.
  Implemented Sudo for dedciated functions. The BoKs alternatieve failed in a technical async terminal line quirk.
• Not getting alignment with external suppliers and changing internal politics.

Release management (versioning)

Within the Information technology guidelines and technics are evolving fast. Recent hyping tools like GIT are getting the most attention.
A generic DTAP Approach DTAP - seeing the leveled three layers is far more important.

Being in a silo you are having just one layer, your own layer.
Doing middleware support you are seeing three layers.
📚 Layered DTAP Release management

• Business Applications Logic
• Middleware tolling like SAS DBMS or a managed filetransfer (MFT)
• Business Applications Logic

🎭 Used technology don´t really matter as long goal release management be met.
💣 That goal alignment being experienced in practice as a problem because the goal often got lost and replaced by a tool implementation.
⚙ Having done and being involved with:

• A well set up DTAP environment, always required scripting (some language).
• The approvement steps are organisational mostly being managed with other tools (ITIL).
  TCAB, DCAB Technical Decision Change Advisory Boards.
• Impact of changes by other layers giving unexplained surprised.

DWH, BI, Data connections, Unix

In a growing enviroment this topics became the only working area.
The first problem to be solved was a generic desktop roll out for SAS cliënts as the desktop got another standard.
The next one was adding and consolidation of midrange servers using SAS (see hosting *AAS stacks). This was a Unix environment (AIX) using a SAN (not NAS). Is has not very much differences in the approach compared to Linux. It makes to set of having experiences with operating systems complete.

📚 Supporting DWH BI Goals:

• Supporting Several business lines not completely abanding the relatively small origin. At the small origin also supporting actuary, marketng (CI) and more.
• Doing the complete release lifecycle support at several layers.
• Getting alignment into security policies (risk managment).

🎭 Used technology:

• AIX, Windows, Mainframes and all dedicated scriptings
• SAS 8.2, SAS 9.1.3 with several solution lines. SAS/intrnet SAS/AF
• Oracle, DB/2, Sql server SSAS Oros, Samba
• ITIL support tools, schedulers (SAS-WA LSF).

⚙ Implemented:

• With the many siloed approaches, went to a hosting approach, consolidating many old implementations. The ODS and Dwh's being included.
• Made all kinds of DBMS connections workable and available at business users.
• Changed the SAS Version Version SAS on all systems.
• Conversion of the manual (SAS macr´s) build schedule flow SAS/WA.
• Stuck in aking SAS datasets behave like anay other DBMS (SAS/Share).
• Solved unexplaianble errors like the one being caused by the old 16Mb line in memory setting and the single to multithreaded order result effects (2010).

Policies - Sox-404 ITSM Cobit IEC/ISO 27001, 27002 - GDPR

Policies standards are becoming mandatory (legal requirement), but are having a lot of documental work to do.

You need to know their goals and than how to get to implement those. That is the top down way.
The technology is having a history of doing hings some way. External suppliers have their own "best practices", sometimes they are very bad practices. That is a bottom up way.
There is need for agreement and archiving on what and how to do it in the own organisation.

The mentioned guidelines are starting points thre are many more of them.
📚 Goals

• Get technical realisations aligned with high level guidelines.
• Following or initiating adjustments on the internal policies.
  Waivers are exceptions declaring not following internal policies.
• Within technical layers having those configured and used conforming those internl policies.

🎭 It is about compliant processes, not the details in technology.
⚙ Having worked on:

• Getting the hands on those high level guidelines.
• Documenting (OSG) and giving feed back on those internal policies.
  experienced feedback audit reports.
• Not always succeful in getting cooperation at other lower technical layers being in control elsewhere.

Data mining, Customer Intelligence (CI)

Data mining, data science is hyping in 2016. The department CI was in my early years (1990´s) one of the busienss lines to support with tools.
Cross selling, customer segmentation, churn rate and more are words they are using. The developmnent being indicated with the word modeling, operational usage of a model using the word scoring. That is using another language to communicate for known processes.

Marketing. Customer Intelligence is commonly using more data sources than are available internally. Geo locations, external open and closed data for inputprocessing bringing into correlations with internal business processes.
Bringing these marketing operations into departments executing the normal classic mass operations is a challenge. The "Analytics Life Cycle" (ALC) is not settled yet.

Customer Intelligence (CI), data mining:
📚 Goals

• The coding tool SAS Eguide being more used instead of classic SAS desktop.
• The low coding tool SAS/EM (Enterprise Miner) getting supported.
• Availabilty of dedicated external data delevivery flows.

🎭 Used technology:

• From early years still runing the SAS 8.2 migrating to SAS 9.3. Adding SAS/EM connecting to DBMS systems (Oracle)

⚙ Having done:

• Solved configuration limits on memory threading and total maximum number of workspace sessions to run at the same moment.
• Seaching connection to PMML standards. Doing an EM course myself.
• Archiving and restoring of EM projects made possible.

ML, Machine Learning, Scoring, Explainable AI (I)

The environment is still sensitive information (2016). A generic description.
📚 Goals:

• Operationalize an existing scoring model that has run manually for a long period manually.
• The Production environment being direct connected for streaming data (message oriented) as input and delivering the results back within that same daily run.
• Delivery to several other production processes with time limits.
• Replace the several old local build solutions within business department with this new approach.

🎭 Used technology used and limitations:

• Process the scoring with a limited SAS tooling set: No availablity of a scheduler, no release management tools, no versioning tools, no segregated environments.
• The executing window is maximum two hours, the whole proces should run in about an hour to serve solving possible incidents.
• The source data is build up in using multiple flows, several windows:
  1. weekly proces that is running severals days to complete. Followed by four hours additonal preparations
  2. Daily updates to complete the too long running weekly proces
  3. Daily changes as feedback of deliverd scores and feedback of changed adjustment within the production proceedings
• The number of cases to be processed as message delivery can be 300.000 (quarterly peak). The total number of cases apx 2.000.000

⚙ The solution design and realization:

• Build and used a manual coded scheduler in SAS that is having all necessary functionality like triggering on events and monitoring on unexpected behavior.
• The release management issue bypassed in an approach with segregation in duties.
• Achieved a very stable hands off process that could be easily transferred to others.

Grid computing - performance load balance

This is a hot topic for performance reasons of business solutions.
The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software Grid computingframe (Herb Sutter) Applications will increasingly need to be concurrent if they want to fully exploit continuing exponential CPU throughput gains Efficiency and performance optimization will get more, not less, important
Scaling up:
📚 Goals:

• Improved performance optimizing usage bare iron.

🎭 Used technologies:

• SAS. linux shell scripting.

⚙ Solutions and realization:

• Gettting annoyed by bad performance as of performance Linux inodes. Bypassed with planned cleanups and diiferent way of coding, record update SAS changed to bulk updates dataset.
• Seeing basic tuning issues once solved at mainframes occuring again. Bypassed with coding and scheduling avoiding IO overload.

Scaling out:
📚 Goals:

• Performance gains by parallel processing usage.

🎭 Used technologies:

• SAS, Teradata (parallel dbms), linux shell scripting.

⚙ Solutions design and realization:

• Using parallele databases like Teradata. Design thinking as little as possible datatransfer between machines instead as ease of coding
• Defining scheduling strategies using multiple machines in cooperation of one execution flow .

ML, Machine Learning, Scoring, Explainable AI (II)

The environment is sensitive information. I can share some approach details.
📚 Goals:

• Operationalize scoring of models in a full DTAP approach
• Improve the data processing so changes can be applied more easily
• One country is aoperational, two others should run similar.
• New Score models should be able adding in a easy way (being agile).

🎭 Used technology used and limitations:

• Scoring is done in a SAS environment (base 9.4). LSF scheduler is available, DI as data integration tool available (ETL ELT data lineage), manual coding (Eguide) is possible.
• Scores should reflect reliable the current situation as near as real-time with the known information. Incremental updating for data changes is every 15 minutes
• There are just two physical machines but a logical four DTAP logical segregation has been defined (develop machine). The production processed aimes to be on one fully isolated machine.
• Information source is gathered in another system (IBM iseries). Data should be synchronized for up to 7.000.000 objects as fact (2.000.0000 active) in an ER-relationship with several dimensions.

⚙ The solution design and realization:

• Made the data synchronisation to a full daily extract from te data source. The question on what could be going missed by "Change Data Capture" avoided. The executing time being within 15 minutes needed to be agreed on network transfer load.
• Got aware of parallel development / test situation. Modelling scores is dependent on data delivery. The score delivery is dependent on scoring. Changed to a set up achieved something workable.
• The releasemangement set up with a acceptance (integration, system test), a shadow production (acceptance and DR) before production.
• The modelling mind switching to ER instead 3nf dimensional.
• Prepared the versioning and releasing using git and bitbucket.
• Added monitoring on the data change process and score process to stop and get notifications when there are an unexpected number of changes. It got the nickname "watch dog"

XML messages, object life cycle data model

Data modelling is a confusing challenge. The requirement is undertanding how information is delivered and what information is needed. These can be complete differnt worlds of context.

The classic dwh approach is based on modelling very detailed elements optimizing the transactional dtabase proces and saving as much as posible storage. The disadvantage is the complexity in relationships.

Blockchain is a hyping approach (2018) for archiving and processing al detailed information and all history in chained blocks (ledger).

In practice contracts, legal agreements, are describing fully the most recent situation. Their history is only in special cases relevant. Those special cases are the most interesting ones when a goal is able to detect illegal activity or fraud.

Use case "real estate":
📚 Goals

• Changed delivery: from database export to XML message driven one.
• The size is 10.000.000 objects as snapshot on dedicated moment. Updates originating from legal action are up to 10.000 daily, but with technical changens that can grow above 100.000 daily.
• Every message is having full details up to 70 record types. Some record types allow repeatitions of 10.000.
• Every message is having the "current" and "previcous" situation with a unique key-reference creating a chain.

🎭 Used technology used and limitations:

• SAS, Linux scripting, Teradata, XML, modernized SQL (XMLTable)

⚙ The solution design and realization:

• Preprocessing splitting XML and transforming it, is necessary.
• A split up in a tremendous lot of small pieces processed did run.
• Due to complexity and some oter issues this could not be transferred to orher persons.

Dwh & datalakes, monitoring, Privacy by design

Storing objects for use at some later moment is warehousing. Just collecting a lot of things not knowing whether you will used it, is another approach.
⚖ The datawarehouse (Bill Inmon 1970, Ralph Kimball 1996) is not having the same goal and same functionality as a physical datawarehouse. In essence, the data warehousing concept was intended to provide an architectural model for the flow of data from operational systems to decision support environments.
What it is describing is the fucntional equivalent of a quality assurance laboratorium and / or a research laboratorimum.
💣 The words "datawarehouse" and "data lake" are confusing in their associations with their physical counterparts

Using the datawarehouse, data lake in line with the physical counterpart.
📚 Goal - change in thinking

• Mulitple customers sharing the same service.
• Solving alignment timely availablity. Allowing JIT processing (lean)
• Allowing delivery of information being used operational when data is originating from a datawarehouse.

🎭   ⚙ Solution and realization :

• The datawarehouse is not a single point dedicated solution:
  1. Several technical implementations possible are coexisting.
  2. bypassing DWH-storage, streaming data Lambda architecture
• The datawarehouse serving several usage types:
  1. Operational processing using tools clasified as BI-tools. The common sense reasosn is the best tool-fit for business requirement.
  2. The classic BI reporting usage (what has happended)
  3. AI, Machine Learning (ML) being devided in two separated flows. The development of models having "privacy by design" restrictions.
     The operational part adding "monitoring " and "evaluation " requirements.

The analytics usage BI (Business Intelligence) descriptive, ML (Machine Learing) and others are just customers of a warehouse like any other operational usage. Being another level of "How to do things" these are the not visible parts in running those other projects.

Patterns

Patterns are basic building blocks and shoud match some issue to solve.
While working on several projects there is a statement "out of the box" predifined transforms are mandatory used.

In reality those "out of the box" steps are lacking some really essential logic (2017).

Several examples of not well suited "out of the box" transforms:
📚 Logical Issues to solve:

• Data synchronisation quality monitoring.
• Scoring processing monitoring on impacted cases (changes).
• 3nf tabel conversion to a DNF dataset.

🎭 Used technology: SAS DI (ETL ELT tool) but generic designthinking.
⚙ Solutions, user generated transforms:

• The issues for data synchronisation are caused by
  1. Numbers processed records to be monitored by a pre - and post process are not generic.
  2. Defining an isolated destination tabel for the monitoring data is not generic.
  3. Defining to logic for generating dynamic SQL for a full or incremental process and the used key selection is not generic.
• The issues for data synchronisation is caused by the resulting scoring result have no generic monitoring definition.
• The 3nf to a dnf transform can be solved with al lot of steps. This is a complicated approach. Another disadvantage is all the performance overhead of all those steps and choosing a not well suited transform even more performance penalties.
  Made a proposal to do this simplified depending on the used database technology.
  Verified these with a known proces did behave well but also showed a system that seems to behave unpredictable (overloaded thrasing).

External references - bookmarks

Most references are made at the main pages of each topic.
Some things in the news that get my attention I put seperately here.
Note: These links will open in a new page.

link , newstopic interest
A Brief History of Machine Learning Keith D. Foote (2019 mar)
Understanding algorithmic decision-making: Opportunities and challenges Scientific Foresight Unit (STOA) (2019 mar)
The first step in AI might surprise you Cassie Kozyrkov (2018 oct)

Most recent updates: design & concepts area

Date	update description
2019wk17	First version on sdlc
2019wk16	First version on bpm

I will not make al long list of this one.

My goal is to have it as stable as possible, when ready.

👓 devops sense
Table of contents, notes updates

Most recent Updates: generic index & devops area

Date	update description
2019wk16	Sense:Acting as a Table of contents, updates, notes
2019wk16	First version on bpm
2019wk12	Main page serving for referencing content redesignd and all minor area´ss prepared with dummy content
2019wk12	JST page (work related 90'´s) converted from old site as technical proof of concept

unhappy on what is going on

Sometimes you get a feeling there is some path still to go, you are not at the real destination yet.

Not really aware what it is until gettting some hints and a little bit being hurt.

Summarize - Explanations

Only a summarize is given here, the explanations are the many detailed documented topics. Any elucidation when needed can be done on request. As an illustration can say much more than a lot of words, I included many illustrations.

Naming Conventions

Having well defined set of names associatied with their types and meaning (metadata) is making a technical realization easy. It would be even far better having a generic standard approach for that.

It is something I have never seen an attempt within the ICT. Document systems: archiving, connecting to busienss workflows are existing. Why don´t they exist in the software development workflow?

Starting a local attempt:
📚 Naming standard like ibrarians have a succesfull approach. Includes the Intial end End of life stages with versioning
⚠ As those namings conventions aren´t generic, porting the solution to another environment is almost impossible. 🎭 Changing a technology can be a business requirement.
⚙ The implementations can be continued without a generic nameing convention using an in house local one that based on tools that are convenient available.

Software Life Cycle mangement

Every enviroment has his omn approach on the Life Cycle Management of hardware, operating system, tools - middleware , business applications. The confusion on the meaning of "the application" is mentioned seperately. The consequence is that is impacting the release management in several ways

There are many dependicies between those .
⚠ Changing the OS layer can impact the behavior of tools - middleware causing unexpected undesirable results at the business applications

⚠ Changing the tools - middleware layer can cause unexpected undesirable results at the business applications

⚠ Needing new features within tools middelware for development for the business it would the best thing to do to change the acceptance (regression testing) and production environmet as first steps after the proof of the installation setup in a infrastructural environment. That is not the Develop Test Acceptance Production order of the business applications.

Security (business: logic - data)

Risk management and security policies are not very well accepted activities as strategy and tactic. They are however high level business requirements by regulations and can´t be ignored

Security of the assets of the business is too often seen as something technical where no accountability exist within the business
💣 Actions by the ICT department pretending to be in security control
⚒ Make the business responsible for the security guideline. See ICT staff as enablers data processor not as data controller (GDPR)

When doing user access management instead of Information Access Management (IAM) the access of business assets by service - and high privileged accounts are getting lost
💣 Important business information assets are left vulnerable at the technology layer in many ways.
⚒ Design and implement security access for tools middleware. Dot that in alignment with the well know user access management approach.

The meaning of "The Application"

What does somebody really meain by the word "the application". The answer is in Who onWhat When. For most people a computer application is just a black box offering results on input.

Within ICT it is less obvious, we have:

1. hardware (physical components, the box)
2. operating system (being programs, code)
3. tools - middleware (being programs, code)
4. business logic (being programs, code) - data (information)

The word is an adjective on what is used on. Each of the higher order one can be seen as being an application of the former.

Confusion is problematic
💣 Hardware, operating person, code (software) else: "the application".
Business person "the application": that is only the business logic - data.
It does not go well for the tools, middelware.
⚒ Use different indicators and words for the layers: business logic - data. and tools middleware

Data governance

Business logic (code) and (information) are important assets. They are to be governed, data governance for the information. The operational area is a well known area. The challenge is in improving operational business processes using analytics.

Process Innovation, stable operations
💣 Seeing the operational environment completely isolated from innovation request underpinned by analytics will result in blocking processes and/or innovations.
⚒ See the data warehouse data lake as their physical representations. A solution for in time availablity of resources in the logistical chain.
⌛ With "just in time" JIT delivery there will be no need for storing. The concept is Streaming data (lambda architecture).
⚖ Copying information to another location is called "datawarehousing" modernized "data lakes". Historical there is no security concept access included, although is should be present.
🎯 The datawarehouse concept is based on the assumption NO information will brought back to operations. That makes no sense (see dwh data lake).

Work Evolution

Preparing data running analyses as requested by researchers. Some analyses has to ran more than once as the insights on the meaning of the data increased. The review and evaluations executed by staticians that is was all done correctly.

When that job would be recruited and published these days it would be named as "data science" and being described with engineering.

The work proces was improved (PDCA kanban) learning from what was expected by requests. When thinking more about that got te conscience that is almost te same steps als was done at the early years when SDM (System Development Methodology, Cap gemini), was promoted.

ICT proces Life Cycle

I have made this recently (beginning 2018).
A closer look and changing words It could be my first job (health research) and the involvment of the first projects at Victoria Vesta (Insurance company) .
👓 There is a link at the cycle illustration.