JST JCL Mainframe

🎭 index references    elucidation    metier 🎭
👐 top    mid    bottom   👐

⚙  bpm   sdlc   bianl   data   meta   math   ⚙
⚒   Intro   Tech (I)   Tech (II)   Initial tasks   Advanced tasks   Results evaluate   ⚒
⚒   Basic Job   Modulair Job   Support Tools   Subsystems   hands free   Realisations evaluate   ⚒

Job Submit Tool, Test automatization - JCL structuring

Making work of ICT colleagues easier.

working with colleagues, full circle dataflow Control testautomation in the releasetrain Control releasetrain Control Data information security Control dataflow Working experiences by some controls. The SIAR cycle is there using the corners 👓, topic cycle in the center.

S Situation
I Initiatives
A Actions
R Realisations

🔰 lost here, than.. devops sdlc.



Reference Topic Squad
Intro Making work of ICT colleagues easier. 01.01
Tech (I) Learning the technical environment (I). 02.01
Tech (II) Learning the technical environment (II). 03.01
Initial tasks Initial Tasks optimised & automated. 04.01
Advanced tasks Advanced Tasks optimised & automated. 05.02
Results evaluate Evaluating results - technical. 06.01
Words of thank to my former colleagues 06.02
Basic Job Standard JCL JOB procedures. 51.01
Modulair Job Standard JCL JOB procedures (continued). 52.01
Support Tools DTA support procedures. 53.01
Subsystems DTA support procedures advanced 54.01
hands free Running automated - hands free 55.01
Realisations evaluate Evaluating realistions. 56.01

sea log

Learning the technical environment (I).

For the processes the limitations of what is technically available at a moment is important. Although the technical situation is not important for the concepts of optimising, they are defining the options in realisations.

The mainframe era
As an example of analysing and solving an ICT problem in a software building process it is still nice. In this case the way of structuring coding at testing and the organizational impact when it evolved.
old days computer
Not an isolated problem is to solve, there are more in this rather simple question. Job submitting, Development Life Cycle, Technical tools availability.

The IBM mainframe approach isn´t that major leading ICT solution suppliers anymore as in the 70´s. Better, cheaper options have been introduced since those early years.

JCL (Job Control Language)
JCL is the script language coding tool in a classic IBM Mainframe used to run processes for the business. JCL can also be used for a lot of other logic to run. JCL is just a coding language. Before you can do anything to optimize the JCL process running business processes you must structure the way of how JCL is coded.
hollerith oldtime
It has a huge advantage in disconnecting the physical data reference out of source code to this JCL storage location (punch cards).

What you do by that is defining which data should be used on the moment the job is run, JIT (Just In Time) definitions.
Any business software, logical program, can be reused in the DTAP setting without having being modified. Avoiding not necessary modifications in business software (production, acceptance) is a quality and compliancy requirement.

Situation found with JCL
In the old previous days everything was on Hollerith cards. After migration those to store on DASD system eliminating these physical artifacts, nothing had really changed. Just in some rare cases a more modern approach has been implemented.
Putting a bunch of a Hollerith cards into a mechanical reader for running something is hard manual work and limited by the paper weight.

The next step was trying to forget the limitations of those old physical cards.
  1. Sorted numbering in positions 73-80 is not needed anymore. The risk of physical dropping it out of your hands is eliminated.
  2. No need to build a complete card-deck as you can get JCL-modules easily from multiple digital datasets

green logl

Learning the technical environment (II).

Users of any kind including ICT staff once were not allowed to operate machines. Operators did have the exclusive rights to do that. This all changed when terminals became normal at the office.

Terminal Interface, computer access
old days computer terminal These kind of terminals were coax attached to a terminal control unit. They were big and clumsy and expensive at the same time very wanted. Before this there wo no options to have access to computer systems.

All involved departments and people did have access to the computer system this way in those days.

old days computer terminal The dark background has become very modern again recent years (2020). The limitation was that number of lines * columns was limited, often used sizes were: 80*24 and 132*27.

This screen is an example how TSO did look like. The user interaction for any application had to be intuitive according to user expectations in the same approach. Any modification to this kind of tailored menu had to be organised in a similar way of installing applications.

Situation found: Roscoe, TSO, Rexx, JES2, IDMS
Roscoe was the multiuser single adresspace product that was in place for coding and development used by ICT staf inlcuding testing and operations support.
The advantage was a relative limited system resource usage.
It had disadvantages: The decision was made to use TSO/ISPF (Time Sharing Option / Interactive System Productivity Facility) for all ICT staff while implementing central security using ACF2. ISPF is offering Screen user interfaces and Rexx became available as language to complete the options for user customisation. One of the possible tasks in TSO is submitting JCL to background jobs.

Background jobs running in job classes and Online systems started in job classes are in a well tuned mainframe system no problem. The workload manager in the system is able to prioritize resource usage in a way critical systems can get the resources they needed. Jes2 is then name of the environment handling the job classes and the JCL jobs submitted.
The interactive business applications are build in IDMS a network DBMS with an integrated data dictionary (IDD) and applications development system (ADS VVTS). Testers reviewing these applications are having terminals for that.

Checklist required tools automated JOB Processing:
  1. ⚙ TSO/ISPF with REXX are able to submit jobs. A batch job can also start another batch job (same submit command or by using the internal reader.
  2. 🎭 Output & log of jobs can be reviewed using TSO/ISPF with SDSF.
  3. 📚 Terminals accessing the computer system are present.
  4. ⚖ Autorisations to all needed actions able to orgnise.
All necessary functionality and user interfaces are present to optimise the daily floor work processes. The only thing left is really doing it.

modern5 logl

Initial Tasks optimised & automated.

It al started when seeing all the manual work, the long time waiting somebody completed his part of work, the same work being repeated over and over again. That repetition has historical grown because of it was done in several departments not having thoughts on cooperation.

JCL JOB guidance Form (JBF)
Found this form as guidance for workers and then got these questions:
  1. Why?
  2. Possible improvements?
  3. Who can help?
  4. What can I do?
  5. When to do it?
The answers for myself to start change by taking the initiative:
  1. It was historical grown
  2. remove the form by trying to automate in what is in there
  3. Allowance of managers team leads at involved departments
    cooperation with workers at those departments fro changes
  4. I could build a tool for submitting those jobs.
  5. Just start and do what is acceptable to most of te involved persons

ToC Theory of constraints clarke ching
The often used jobs for testers (Pareto)
I got a limited list of often used test jobs.
✅ Build front end screen (ISPF panel) for each of then. PTG could enter the wanted data in, using some default setting to limit the amount of typing.
✅ The JCL for the jobs went to a skeleton (ISPF) dataset option. That was documented by IBM in ISPF documentation to generate data as JCL-jobs in an example.
---🤔--- Disadvantage: every job a unique set in panel, skeleton.
🎯 The result was that PTG could do more than a half of the jobs themselves abandoning most of the hardcopy paper work.

Structuring JCL, a requirement along with operations (JCL2000)
We got time to do a conversion off all test jobs.
✅ Copied all existing jobs to the a skeleton library. Stripping the header (jobname program location search). Changed fixed datasetnames into avariable name.
✅ Moved IO (input output definitions) into an "include lib".
✅ Made a new front end wiht panels (screens) allowing for the more generic appraoch. Saving what was has been input by PTG in an ISPF-table.
✅ Added in a trailer step what job to start when everything went right (cc=0).
🎯 The result was that PTG could do almost all of the jobs themselves.
      Reruns became very easy. Abandoning the hardcopy paper work.

All kind of batch jobs for testers additional tools (compare)
The questions could all be solved with available tools, software:
✅ Build in the backup restore program at several points and as dedicated tool.
✅ Made result of several jobs able to store in datasets.
✅ Build in compare options as dedicated tool.
🎯 The result more test-jobs could run without any additional human constraints.
🎯 PB (OPS) could copy wat has done at test. The JBF paper form abandoned.
Handy tool
The tools were described separately.
Comparing with the TSO tool was advanced with a lot of options what to include and what not.
Backup and restore needed to be aligned with naming conventions of the datasets.

The DTAP structure wiht naming convetions was not that evident. The problem is that the FLQ (first level qualifier) was based on department names. The Second level qualifier very distinctive for business logic, data, input logic etc.

modern7 logl

Advanced Tasks optimised & automated.

The more got automated in the test, operations and development, more functionality was built in. Education of people being an options in user acceptance testing using the operational code with selected test data. Technical software changes is a similar way having several versions of the tools middleware on the same machine.

On line Transaction Processing (IDMS)
The question of the OLTP support triggred another usage type of the DBMS including the frontend (OLTP). The education center for people did also use those OLTP systems wiht the differnce that the had production code.

Two series for OLTP systems tot start were defined:
✅ Defining the database to be used and the OLTP system number for parallel usage of an OLTP interface.
✅ Having the OLTP waited for the database availablity before continuing.
✅ Having simulated terminal input for the OLTP system settings waited until the OLTP system is responsive.
✅ Having simulated terminal input with business logic waiting until the OLTP system setting are finished. Continue with saved user sessions and user interactions.
🎯 The result a self service usage with test and education for OLTP usage.

Release mangement, parallel testing development (Endevor)
First versions JST aligned to DBA (OLTP versions IDD) and PB home grown tools.
ToC Theory of constraints clarke ching The IDMS IDD needed an additional plug-in that was not common in use.
All OLTP programs had to move from internal IDMS (DBMS) storage to external system storage.
Naming conventions of used datasets having programs needed to change.
A lot of user modifications named endevor processes to build.

Two series for OLTP systems tot start were defined:
✅ All names of program libraries changed.
✅ For parallel development additional names storage and IDD´s were defined.
🎯 The mismatch in expectations caused DEV to use also the JST approach.
      The full DTAP line under control of release management and able to run tests.
🎯 Also PB went to use it more for preproduction and acceptance testing.

Time travelling, unit conversions (Milo Euro)
The additional tool was added for easier time travelling:
✅ Options for shifting time interval added.
🎯 Tests for milo and euro very extensively done, no issued at those change moments.
Time travelling with data was a lot more complex than expected. The day of week and holiday indications are never getting similar.< Worse was that some date fields were used with an invalid date because the functional management did not know of better options to have an indication of a system change.

Printing Services, IO guidelines
For printing using form layouts and signatures:
✅ additional statements for technology in use in the JCL header defined and maintained.
🎯 The best option I could think of to isolate this special technology.
      I would have wished there was a better security and more proper test data.


Evaluating results - technical.

The main technical issue was structuring JCL so it could be automated and reused. The tool it self was build with TSO/ISPF using REXX at no additional cost with those common available tools.

Naming conventions and standard way of work is the key issue. Not easily copies into another organisation.

Used concepts & practical knowledge
Reconsidering this nice project, it was nice because it has that many connections to a diverse set of activities. It serves to connect all those topics.
detailed descriptive information at ..
Details to be found at:
👓 Software Life Cycle (business)..
👓 Security ..
👓 Data modelling ..
👓 EIS Business Intelligence & Analytics ..
👓 Tools Life Cycle (infrastructure)..
👓 Low code data driven work as design..

The security association may not that obvious.
Simulating the production using test-accounts, in a way they are like operational production usage, is a good practice.
Having the overall security and configuration in a way you can run a complete DTAP on a single box (Iron) having containerized anything, is a good practice.

Words of thank to my former colleagues
We had a lot fun in stressfull times working out this on the mainframe in the nineties. The uniquenes and difficulties to port it to somewhere else realising.
A few names: Many more people to mention, sorry when I didn´t. (notes: 2008)

Not always unhappy by what is going on
Ending bridge
this topic ends here. back to devops sdlc
👓 🔰
⚒   Intro   Tech (I)   Tech (II)   Initial tasks   Advanced tasks   Results evaluate   ⚒
⚒   Basic Job   Modulair Job   Support Tools   Subsystems   hands free   Realisations evaluate   ⚒

Implementing frames.

Standard JCL JOB procedures.

Goals Automated JOB Processing :
  1. Automated testing: regression approach and new logic or fixing found issues
  2. preparing the data needed for tests
  3. preparing jobs / scripts in a low code approach
  4. Automated running: all the jobs / scripts
  5. Saving & archiving & comparing the results.
Not all was don at the same moment, changes and functionality grew incremental.

First basic conversion to automatization
Structured scripting
header trailer Defining all globals, job environment settings.
Bodies - Frames
body frame Business logic software application programs.
Data Definitions
ddio1 ddio2 ddio3 Input, Output, Report -log for every step.
header trailer Defining actions, jobs after this one.
Head :

You need somehow to define the datasets output / print / mail / databases / transfers to be used. These definitions may not be hard coded in the business logic code. Why not?
Tail :

Standard JCL JOB procedures (continued).

Bodies - Frames (Unstructured) :
This will work and will be manageable when having not that many jobs to run. A next improvement is structuring this. Define steps conforming business logic using standard building patterns. Support by functional and technical owners infrastructure middleware components is needed.

Normalised basic conversion to automatization
Structured scripting
Common body / frame
body frame Used by business functional owners.
Common proc bodies
procedure body Technicals you can reuse very often.
These technicals are a limited number of JCL proc steps, like: A modulair proc step in this way is having api (program interface) options.
The normalising of the data into segregated components is creating new artefacts.

New artefacts, objects naming conventions
The word local is used in context to the machine where the process is executed. Not necessary the same location as where the human worker is.

Standard user defined values requirement:
  1. Unique data identification (database, datasets, jobs)
    reason: Allow to do tests in parallel.
  2. Data environment indicator.
    One of the well known DTAP (Development Test Acceptance Production).
  3. Application code environment inidcatir.
    One of the well known DTAP (Development Test Acceptance Production).
  4. Application code parallel development when applicable.
    reason: Allow to do devlopment in parallel.
  5. Date indicator the application should use instead of the system date.
    Underpinning: A process although designed on the current date should be able to run on an other date.

DTA support procedures.

Other procedures needed are :

Procedures as additional needs
Structured scripting backup restore
Backup Restore script D, T, A,
Backup & restore is essential to D,T,A Develop & Test, Acceptance environments. The test environment results are important business assets.

Evidence of quality (auditable).

body frame Used by Testers.

To allow many testers to work parallel, segregation in naming and environment should be possible. Verifying content of backups must be possible.

Support by functional - technical owners infrastructure components.

A Acceptance tests differentations
DTA Develop, Test, Acceprance testing are phases before it goes into operations (Production).
Acceptance testing is with more variations as is visible in the SDLC reference.
Activities can have origins in project planning, for example education of humans.
Within the operations business requirements of continuity can get a solution.

Within A (acceptance) stage the business data can be different to it´s usage goal, as there are: Education is more eassy using a selected set (small data) conform Test (PTG).

Backup & Restore, Performance experiences
The backup and restore of a complete test plan used standard backup & restore utilities. These are optimize for that work collecting and compressing everything as first stage. Next stage writing that optimized for a tape device doing sequential IO.
💣 When getting forced not to use those tools found a big performance penalty because of locking queuing on shared DASD is much slower.
It is not what is told as dasd is promoted to be faster.

DTA support procedures advanced

Interactive Transactional Systems
batch - transactions Aside batch processing, interactive systems (transactional), are also part of business information system.
Mainframe workloads: Batch and online transaction processing
Most workloads fall into one of two categories: Batch processing or online transaction processing, which includes Web-based applications.

Integrated Data Dictionary (IDD IDMS Cullinet)
IDD Integrated Data Dictionary - Promoting Management CA IDMS Reference - Software Management
Promotion Management is the process wherein entities are moved from environment to environment within the software development life cycle. When these environments consist of multiple dictionaries, application development typically involves staged promotions of entities from one dictionary to another, such as Test to Quality Assurance to Production. This movement can be in any direction, from a variety of sources.

Structured scripting dedicated dbms
Dedicated DBMS script D, T, A,
This feature essential to D,T,A Test environments when transactional systems are involved.

The difference to backup restore support is: This implementation can be much more challenging is the complexity of start-up DBMS support requires to do so.

Infra-Process transactional. body frame Used by Testers.

Support by functional - technical owners infrastructure components.

Interactive Transactional Systems - Integrated Data Dictionary
These experiences are essential building up my mind. An IDD concept is hot in data governance. The new word is "Metadata".
Same concepts and approaches can be used with all those new tools.
Having multiple IDD's needed for the required DTAP segregations, used was:

Running automated - hands free

Basic design
This looks at little bit complicated, the basics are: There is an option to define a flow of processes to run.

Modelling the JST process
The proces model has several dimensions: Design JST The red arrows in the figure are indicating the optional selections for:
  1. jobs - All the defintions to get them run wiht predefined settings and values.
  2. testplans - The collection of jobs that make a testrun as unit.
  3. The fact in a star-schema ER-relationship is this the: "job".

Running local batch, central version transactional, experiences
Running with a local version of the DBMS is going for locking area´s not locking every reacord (OLTP). Going for locking every record in a transactional way updating infroamtion has the advantages: of updateThese are optimize for that work collecting and compressing everything as first stage. Next stage writing that optimized for a tape device doing sequential IO.
When getting forced not to use those tools found a big performance penalty because of locking queuing on shared DASD is much slower.


Evaluating realistions.

Life cycle JCL transition
The change is a structured JCL generating and usage approach. I designed that in coöperation with others. Splitting up the old Job´s into basic building blocks, reuse them in every job for every type of usage.
When you can generate JCL, it is also possible to generate a lot of different ones with the goal of segregated parallel testing. Starting a next job when one is ready is as easy as a manual start.
💰 The cost saving in time is obvious: doing something once instead the same thing over and over again by different people.
💣 Difficult to get it accepted by ICT staf users. Remember: You are (I am) automating the daily work of some or more persons (ICT staff)

Missing for deploying JST As A Service JST-AAS
Naming Conventions
Life Cycle Management
Security (business: data / code)

Not always unhappy by what is going on
working with colleagues, full circle The releasetrain Control testautomation in the releasetrain Control releasetrain Control Data information security Control dataflow Working experiences by some controls. The SIAR cycle is there using the corners 👓, topic cycle in the center.

S Situation
I Initiatives
A Actions
R Realisations

🔰 lost here, than.. devops sdlc.

⚒   Basic Job   Modulair Job   Support Tools   Subsystems   hands free   Realisations evaluate   ⚒
⚒   Intro   Tech (I)   Tech (II)   Initial tasks   Advanced tasks   Results evaluate   ⚒
⚙   bpm   sdlc   bianl   data   meta   math   ⚙

© 2012,2020 J.A.Karman
👐 top    mid    bottom   👐
🎭 index references    elucidation    metier 🎭