Rechercher

Article
· Sept 17, 2020 4m de lecture

Native API for ObjectScript Demo

The demo is based on the raw class descriptions.
The data classes used are Address, Person, Employee, Company
For a more attractive demo, a JSONtoString method by ID was added.

After installation with ZPM just run from Terminal

USER>do ##class(rcc.ONAPI.demo).Run()
Adjust Parameters
host[127.0.0.1]:
port[51773]:
namespace[USER]:
user[_SYSTEM]:
pwd[SYS]:
timeout[5]:
****** connected ********

Next, you get a list of possible demo actions.
No input means no action.
The menu loops until you exit.

Populate Person by:100
     100
Populate Company by:10
     10
Populate Employee by:50
     50
Show Person by ID:3
     {"Name":"Rogers,Norbert V.","SSN":"990-11-9806","DOB":"1962-04-23","Home":{"Street":"867 Oak Street","City":"Denver","State":"NH","Zip":"64647"},"Office":{"Street":"3309 Oak Court","City":"Denver","State":"NY","Zip":"76436"},"FavoriteColors":["Green","Green"],"Age":58}

Show Company by ID:3
     {"Name":"CompuComp Corp.","Mission":"Specializing in the development and manufacturing of open-source object-oriented models for additive manufacturing.","TaxID":"Y8155","Revenue":819493934,"Employees":[{"Name":"Ahmed,Sophia P.","SSN":"936-73-5161","DOB":"1933-08-08","Home":{"Street":"8758 Elm Street","City":"Fargo","State":"OH","Zip":"40652"},"Office":{"Street":"1578 Maple Street","City":"Larchmont","State":"IA","Zip":"89021"},"Spouse":{"Name":"Olsen,William A.","SSN":"912-52-4809","DOB":"2010-01-26","Home":{"Street":"2933 Main Street","City":"Bensonhurst","State":"WA","Zip":"51960"},"Office":{"Street":"4994 Ash Street","City":"Gansevoort","State":"OR","Zip":"89750"},"Spouse":{"Name":"Adam,Brian D.","SSN":"799-82-3083","DOB":"2005-11-04","Home":{"Street":"1264 Oak Avenue","City":"Chicago","State":"WV","Zip":"34943"},"Office":{"Street":"7443 Second Avenue","City":"Zanesville","State":"CO","Zip":"30478"},"Spouse":{"Name":"Cooke,Diane C.","SSN":"754-49-5729","DOB":"1984-01-12","Home":{"Street":"9624 Maple Place","City":"Albany","State":"MS","Zip":"60948"},"Office":{"Street":"4711 Second Place","City":"Youngstown","State":"VA","Zip":"31250"},"Age":36},"FavoriteColors":["Orange"],"Age":14},"FavoriteColors":["Green","Green"],"Age":10,"Title":"Senior Systems Engineer","Salary":61511},"Age":87,"Title":"Product Manager","Salary":59127},{"Name":"Ng,Elmo S.","SSN":"201-15-3259","DOB":"1954-10-14","Home":{"Street":"2437 Main Avenue","City":"Xavier","State":"KY","Zip":"10156"},"Office":{"Street":"2770 Oak Drive","City":"Tampa","State":"OH","Zip":"18459"},"Spouse":{"Name":"Eastman,Linda M.","SSN":"110-41-1818","DOB":"1980-03-19","Home":{"Street":"5309 Ash Drive","City":"Xavier","State":"RI","Zip":"36964"},"Office":{"Street":"4288 Washington Place","City":"Xavier","State":"HI","Zip":"41889"},"Spouse":{"Name":"Huff,Dave C.","SSN":"559-32-3838","DOB":"1973-05-05","Home":{"Street":"6216 First Avenue","City":"Tampa","State":"WA","Zip":"68628"},"Office":{"Street":"2896 Clinton Drive","City":"Elmhurst","State":"UT","Zip":"97796"},"FavoriteColors":["Purple"],"Age":47},"FavoriteColors":["Purple","Blue"],"Age":40},"FavoriteColors":["Red","Purple"],"Age":65,"Title":"Laboratory Marketing Manager","Salary":35888}]}

Show Employee by ID:103
     {"Name":"Faust,Buzz H.","SSN":"979-41-6347","DOB":"1938-02-07","Home":{"Street":"6231 Madison Avenue","City":"Gansevoort","State":"TX","Zip":"49085"},"Office":{"Street":"6402 Main Street","City":"Elmhurst","State":"RI","Zip":"82976"},"Spouse":{"Name":"Joyce,Chad C.","SSN":"199-86-8085","DOB":"1974-11-17","Home":{"Street":"6229 Main Street","City":"Reston","State":"FL","Zip":"16922"},"Office":{"Street":"8509 Elm Blvd","City":"Bensonhurst","State":"HI","Zip":"90665"},"Spouse":{"Name":"Cooke,Diane C.","SSN":"754-49-5729","DOB":"1984-01-12","Home":{"Street":"9624 Maple Place","City":"Albany","State":"MS","Zip":"60948"},"Office":{"Street":"4711 Second Place","City":"Youngstown","State":"VA","Zip":"31250"},"Age":36},"FavoriteColors":["Purple"],"Age":45},"Age":82,"Title":"Global Administrator","Salary":13813}

Show Global PersonD by ID:4
     $Data()=1
     Value=$lb("","Eastman,Mary C.","887-18-3730",44711,$lb("3889 Ash Blvd","Washington","TX",67862),$lb("5709 Oak Blvd","Chicago","IL",30845),"","")

Index list for Person & Employee (n,y):y
     $Employee
     $Person
     NameIDX
     SSNKey
     ZipCode
Exit Demo (n,y,*):
Populate Person by:
Populate Company by:
Populate Employee by:
Show Person by ID:
Show Company by ID:
Show Employee by ID:104
     {"Name":"Novello,Emily I.","SSN":"411-35-4234","DOB":"1943-01-07","Home":{"Street":"3353 Washington Court","City":"Hialeah","State":"MT","Zip":"22403"},"Office":{"Street":"9743 Clinton Blvd","City":"Xavier","State":"OH","Zip":"89038"},"Spouse":{"Name":"Goldman,Usha T.","SSN":"465-59-4053","DOB":"1987-07-16","Home":{"Street":"2578 Second Blvd","City":"Gansevoort","State":"FL","Zip":"77552"},"Office":{"Street":"6986 Main Street","City":"Elmhurst","State":"VT","Zip":"48713"},"Spouse":{"Name":"Rogers,Norbert V.","SSN":"990-11-9806","DOB":"1962-04-23","Home":{"Street":"867 Oak Street","City":"Denver","State":"NH","Zip":"64647"},"Office":{"Street":"3309 Oak Court","City":"Denver","State":"NY","Zip":"76436"},"FavoriteColors":["Green","Green"],"Age":58},"Age":33},"FavoriteColors":["Green","White"],"Age":77,"Title":"Senior Product Specialist","Salary":96469}

Show Global PersonD by ID:
Index list for Person & Employee (n,y):
Exit Demo (n,y,*):y
****** done ********
 
USER>

GitHub

Discussion (0)0
Connectez-vous ou inscrivez-vous pour continuer
Article
· Sept 14, 2020 3m de lecture

IRIS Native API for ObjectScript

It seems to me that for some reason this didn't make its way to the official documentation
and seems to be rather unknown though implemented already in IRIS 2020.1

Thanks to @Dan Pasco I got a hint on the classes involved.
I used the recommended sequence of how to use it. 
it is all directly taken from Class Reference and I just collected it to create a first overview.

The Native API for ObjectScript follows the API used on other language platforms. It starts with a static API that allows the user to connect to an IRIS Namespace, either local or remote, using an Iris Connection. The connection interface allows the user to then instantiate the IRIS Native API class (%Net.DB.Iris). That instance can then be used to access IRIS Global Arrays, manage transactions, invoke functions and methods implemented in routines or classes, and instantiate remote iterators.

%Net.DB.DataSource

This class implements the IRIS Native API for Object Script DataSource interface.
At this time that interface consists solely of the CreateConnection method.

CreateConnection accepts url, port,namespace, user, and pwd parameters. Refer to %Net.DB.Connection for more information on these parameters. CreateConnection() returns an instance of %Net.DB.Connection.

host input The host name or address, defaults to 127.0.0.1
port input The port on which to connect
namespace input The namespace to connect to

user

input The user's name
pwd input User's password
timeout input The number of seconds to wait for the connection to be established. Optional.
logfile input The name of the file to be used for logging. If specified, connection activity will be logged to that file. Optional.
  return An oref referencing an instance of %Net.DB.Connection

 

 %Net.DB.Connection

This class implements the IRIS Native API for Object Script Connection interface.
This class should never be instantiated directly, only through %Net.DB.DataSource using the CreateMethod() function.

The public interface for this class includes only those items specified here. All other members are internal.

Host property The host specified when establishing the connection
Port property The port that this connection is connected.
Namespace property The namespace that this connection is connected.
Timeout property The timeout value specified establishing the connection
IsClosed method Returns true if this connection instance is no longer connected.
Close method Close this connection.
CreateIris method Returns an instance of %Net.DB.Iris.

 

%Net.DB.Iris

This class implements the primary set of functions IRIS Native API for Object Script.
The only other members of the IRIS Native API that are not implemented by this class
are implemented by %Net.DB.DataSource and %Net.DB.Connection.
This class should never be instantiated directly.
The proper way to instantiate this class is to invoke the CreateIris() method using an instance of %Net.DB.Connection.

Summary of methods:

 

GitHub

1 Comment
Discussion (1)0
Connectez-vous ou inscrivez-vous pour continuer
Question
· Sept 11, 2020

Data Drive Web Apps Questions

Hey there,

I posted a reply to the recent video https://community.intersystems.com/post/new-video-building-data-driven-web-apps#comment-132511
with a slew of questions.  Wondering if someone can take a look a address my questions on that post or on this new thread.

I find many InterSystems learning to be challenging to follow, but I"m not sure if it's because my work computer is so locked down that I don't have all the right tools and permissions I need to do what the tutorial asks, or if I'm not understanding the basics of Cache/IRIS, or both!

Mike

2 Comments
Discussion (2)1
Connectez-vous ou inscrivez-vous pour continuer
Article
· Sept 2, 2020 7m de lecture

Integrity Check: Speeding it Up or Slowing it Down

While the integrity of Caché and InterSystems IRIS databases is completely protected from the consequences of system failure, physical storage devices do fail in ways that corrupt the data they store.  For that reason, many sites choose to run regular database integrity checks, particularly in coordination with backups to validate that a given backup could be relied upon in a disaster.  Integrity check may also be acutely needed by the system administrator in response to a disaster involving storage corruption.  Integrity check must read every block of the globals being checked (if not already in buffers), and in an order dictated by the global structure. This takes substantial time, but integrity check is capable of reading as fast as the storage subsystem can sustain.  In some situations, it is desirable to run it in that manner to get results as quickly as possible.  In other situations, integrity check needs to be more conservative to avoid consuming too much of the storage subsystem’s bandwidth. 

Plan of Attack

This following outline caters for most situations.  The detailed discussion in the remainder of this article provides the necessary information to act on any of these, or to derive other courses of action. 

  1. If using Linux and integrity check is slow, see the information below on enabling Asynchronous I/O. 
  2. If integrity check must complete as fast as possible - running in an isolated environment, or because results are needed urgently - use Multi-Process Integrity Check to check multiple globals or databases in parallel.  The number of processes times the number of concurrent asynchronous reads that each process will perform (8 by default, or 1 if using Linux with asynchronous I/O disabled) is the limit on the number of concurrent reads in flight.  Consider that the average may be half that and then compare to the capabilities of the storage subsystem.  For example, with storage striped across 20 drives and the default 8 concurrent reads per process, five or more processes may be needed to capture the full capacity of the storage subsystem (5*8/2=20).
  3. When balancing integrity check speed against its impact on production, first adjust the number of processes in the Multi-Process Integrity Check, then if needed, see the SetAsyncReadBuffers tunable.  See Isolating Integrity Check below for a longer-term solution (and for eliminating false positives).
  4. If already confined to a single process (e.g. there’s one extremely large global or other external constraints) and the speed of integrity check needs adjustment up or down, see the SetAsyncReadBuffers tunable below.

Multi-Process Integrity Check

The general solution to get an integrity check to complete faster (using system resources at a higher rate) is to divide the work among multiple parallel processes.  Some of the integrity check user interfaces and APIs do so, while others use a single process.  Assignment to processes is on a per-global basis, so checking a single global is always done by just one process (versions prior to Caché 2018.1 divided the work by database instead of by global).

The principal API for multi-process integrity check is CheckLIst^Integrity (see documentation for details). It collects the results in a temporary global to be displayed by Display^Integrity. The following is an example checking three databases using five processes. Omitting the database list parameter here checks all databases.

set dblist=$listbuild(“/data/db1/”,”/data/db2/”,”/data/db3/”)
set sc=$$CheckList^Integrity(,dblist,,,5)
do Display^Integrity()
kill ^IRIS.TempIntegrityOutput(+$job)

/* Note: evaluating ‘sc’ above isn’t needed just to display the results, but...
   $system.Status.IsOK(sc) - ran successfully and found no errors
   $system.Status.GetErrorCodes(sc)=$$$ERRORCODE($$$IntegrityCheckErrors) // 267
                           - ran successfully, but found errors.
   Else - a problem may have prevented some portion from running, ‘sc’ may have 
          multiple error codes, one of which may be $$$IntegrityCheckErrors. */

Using CheckLIst^Integrity like this is the most straight-forward way to achieve the level of control that is of interest to us.  The Management Portal interface and the Integrity Check Task (built-in but not scheduled) use multiple processes, but may not offer sufficient control for our purposes.*

Other integrity check interfaces, notably the terminal user interface, ^INTEGRIT or ^Integrity, as well as Silent^Integrity, perform integrity check in a single process. These interfaces, therefore, do not complete the check as fast as it's possible to achieve, and they use fewer resources.  An advantage, though, is that their results are visible, logged to a file or output to the terminal, as each global is checked, and in a well-defined order.

Asynchronous I/O

An integrity check process walks through each pointer block of a global, one at a time, validating each against the contents of the data blocks it points to.  The data blocks are read with asynchronous I/O to keep a number of read requests in flight for the storage subsystem to process, and the validation is performed as each read completes. 

On Linux only, async I/O is effective only in combination with direct I/O, which is not enabled by default until InterSystems IRIS 2020.3.  This accounts for a large number of cases where integrity check takes too long on Linux.  Fortunately, it can be enabled on Cache 2018.1, IRIS 2019.1 and later, by setting wduseasyncio=1 in the [config] section of the .cpf file and restarting.  This parameter is recommended in general for I/O scalability on busy systems and is the default on non-Linux platforms since Caché 2015.2.  Before enabling it, make sure that you’ve configured sufficient memory for database cache (global buffers) because with Direct I/O, the databases will no longer be (redundantly) cached by Linux.  When not enabled, reads done by integrity check complete synchronously and it cannot utilize the storage efficiently. 

On all platforms, the number of reads that an integrity check process will put in flight at one time is set to 8 by default.  If you must alter the rate at which a single integrity check process reads from disk this parameter can be tuned – up to get a single process to complete faster, down to use less storage bandwidth.  Bear in mind that:

  • This parameter applies to each integrity check process.  When multiple processes are used, the number of processes multiplies this number of in-flight reads  Changing the number of parallel integrity check processes has a much larger impact and therefore is usually the first thing to do.  Each process is also limited by computational time (among other things) so there increasing the value of this parameter is limited in its benefit.
  • This only works within the storage subsystem’s capacity to process concurrent reads. Higher values have no benefit if databases are stored on a single local drive, whereas a storage array with striping across dozens of drives can process dozens of reads concurrently.

To adjust this parameter from the %SYS namespace, do SetAsyncReadBuffers^Integrity(value). To see the current value, write $$GetAsyncReadBuffers^Integrity(). The change takes effect when the next global is checked.  The setting currently does not persist through a restart of the system, though it can be added to SYSTEM^%ZSTART.

There is a similar parameter to control the maximum size of each read when blocks are contiguous on disk (or nearly so).  This parameter is less often needed, though systems with high storage latency or databases with larger block sizes could possibly benefit from fine tuning.  The value has units of 64KB, so a value of 1 is 64KB, 4 is 256KB, etc.  0 (the default) lets the system to select and it currently selects 1 (64KB).  The ^Integrity function for this parameter, parallel to those mentioned above, are SetAsyncReadBufferSize and GetAsyncReadBufferSize.

Isolating Integrity Check

Many sites run regular integrity checks directly on the production system. This is certainly the simplest to configure, but it’s not ideal.  In addition to concerns about integrity check’s impact on storage bandwidth, concurrent database update activity can sometimes lead to false positive errors (despite mitigations built into the checking algorithm).  As a result, errors reported from an integrity check run on production need to be evaluated and/or rechecked by an administrator.

Often times, a better option exists.  A storage snapshot or backup image can be mounted on another host, where an isolated Caché or IRIS instance runs the integrity check.  Not only does this prevent any possibility of false positives, but if the storage is also isolated from production, integrity check can be run to fully utilize the storage bandwidth and complete much more quickly.  This approach fits well into the model where integrity check is used to validate backups; a validated backup effectively validates production as of the time the backup was made.  Cloud and virtualization platforms can also make it easier to establish a usable isolated environment from a snapshot.

 


The Management Portal interface, the Integrity Check Task and the IntegrityCheck method of SYS.Database select a rather large number of processes (equal to the number of CPU cores), lacking the control that’s needed in many situations. The management portal and the task also perform a complete recheck of any global that reported error in effort to identify false positives that may have occurred due to concurrent updates. This recheck occurs above and beyond the false positive mitigation built into the integrity check algorithms, and that may be unwanted in some situations due to the additional time it takes (the recheck runs in a single process and checks the entire global). This behavior may be changed in the future.

8 Comments
Discussion (8)0
Connectez-vous ou inscrivez-vous pour continuer
Article
· Août 30, 2020 1m de lecture

How to Restart SMP server

Caused by a conflict in the port assignment I get this  entry in mesages.log and SMP doesn't respond:

08/30/20-12:56:40:714 (15232) 1 [Utility.Event] Private webserver may not start on port 52773, may be in use by another instance
08/30/20-12:56:40:737 (15232) 0 [Utility.Event] Private webserver started on 52773

The first line is true, the second is just wishful thinking sad

demo code on GitHub

6 Comments
Discussion (6)0
Connectez-vous ou inscrivez-vous pour continuer