File based transfers using the integrator

Owned by Marc Dumontier
Last updated: Aug 28, 2017
3 min read

This guide will explain some of the details around the changes in  OSCAREMR-6104 - Getting issue details... STATUS   OSCAREMR-6112 - Getting issue details... STATUS  and detail the integrator (shared record) process.



This new architecture separates the export, transfer, and import of update files from an OSCAR instance to an integrator server.


The basic idea is that OSCAR runs an internal process periodically which:

1) Deternines which records to send (full push vs delta)

2) Writes a set of serialized files and a documentMeta.txt file which are then zipped.

3) Creates a ZIP file of all Documents needed by the import process (see documentMeta.txt)


When the ZIP file is complete, the files can be made available to the integrator to import. Because this is such a system dependent process, it is up to the deployer to develop a process for this. A common way would be to simply scan for IntegratorPush zip files (not the -Docs ones) made by OSCAR and then copy them to the server/directory specified by the importer using any protocol like SCP, SFTP, RSYNC,etc


The integrator is then configured to scan a directory for incoming files. When found, it will attempt to process the file by unzipping it, and beginning to read the serialized files then going on to import the documents into the DB as well.

 


OSCAR

INTEGRATOR_UPDATE_PERIOD_PROPERTIES_KEY is used to know how often to run the job. Regardless it should run pretty much after starting up tomcat. Make sure you configured the Facility (enabled integrator).

You should see all the output in integrator.log not catalina.out

When the job runs, the output will be 2 files

IntegatorPush_<FacilityId>_yyyy_MM_dd_HH_mm_ss.zip

IntegatorPush_<FacilityId>_yyyy_MM_dd_HH_mm_ss-Docs.zip

Where yyyy_MM_dd_HH_mm_ss is the same for both files.


OSCAR will also record this creation event in the IntegratorFileLog table. 


There is also an OSCAR Job (org.oscarehr.PMmodule.caisi_integrator.IntegratorFileLogUpdateJob) which needs to be setup on a schedule as well. This job will update the status in IntegratorFileLog by checking  with the integrator through web services if it has processed (successfully or error) the file. You can see a UI for the IntegratorFileLog table in the new Integrator Status page from the Administration module.


In terms of the serialized files. When you unzip a transfer file, you will see a bunch of entries like


IntegatorPush_<FacilityId>_yyyy_MM_dd_HH_mm_ss.1.ser

IntegatorPush_<FacilityId>_yyyy_MM_dd_HH_mm_ss.2.ser

...

documentMeta.txt


The .ser files are just JAVA serialized objects using the native functionality of JAVA. What's important is that the first file contains a special header object, and the last file will end with a special footer object. The header contains critical information (like the file that needs to be run before this one can be run). The footer is more of a placeholder so the importer knows it successfully finished. Everything else is the data except Document contents.





#set false if you only have 1 facility on this instance..a bit of an optimization.
INTEGRATOR_LIMIT_TO_FACILITY=false


Startup tomcat, you will need to include -Dorg.apache.cxf.stax.allowInsecureParser=1 in your startup command for Tomcat


Integrator

Use JDK7 / Tomcat 7 (atleast I do in my testing to make it work)

Just load up the code with this update

The importer is built in to the integrator..so as it runs, it will scan a chosen directory, and as files are dropped in there that follow the proper naming scheme, it will be loadewd and processed.


It's currently hard coded to run every minute. When a process is running, a lock file is created so that if the next job starts up, it knows not to run concurrently. (importer.lock in TOMCAT tmp directory)

All logging will go to catalina.out


Currently hard coded properties in integrator - it will use /usr/share/tomcat7/IntegratorFiles to scan for files, and archive them /usr/share/tomcat7/IntegratorFiles/archive. It's working directory will be /tmp


you need to have the 'unzip' program installed in the path for tomcat user.





Importing

Make a copy of importer.cnf

mvn assembly:single to create a single JAR with dependencies

java -jar integrator-importer-1.0-SNAPSHOT-jar-with-dependencies.jar to start the scanner