Database Definitions in PDF Form: VeracityUK1.2.pdf (13 MB)

Database Definitions in XPS Form (better for searching): VeracityUK1.2.XPS (2 MB)

SQL Server 2005 Database Backup: ZIAgent2.3.BAK (26 MB)

SQL Server 2005 Database CREATE DB Script: UKAgent1.2script.sql (1 MB)

SQL Server 2005 Database Backup of a database with sample UK 1.1 data: ZIProviderUK.BAK (60 MB)

Introduction

In June 2009, Dell, Visual Software and ITWorx assembled a Proof of Concept for the Cambridgeshire BSF bid and demonstrated that the Schools Interoperability Framework could be used to transport learner information such as that which would come from a typical UK MIS to a learning platform in near real time to meet the requirements of the Local Authority (LA).

In the first phase, for several reasons, we used a MIS simulator instead of using a copy of Capita SIMS, Serco CMIS or one of the other MIS systems.  In the second phase of this overall project, we will be putting together a lasting platform that will be used in the Cambridgeshire and Derbyshire implementations.

This document’s first purpose is to supply Visual Software’s recommendations for setting up a SIF implementation for these schools, connecting the MIS instances, providing matching, identity and role management, providing facilities for checking the data for quality standards and providing an interface for the other Dell-incorporated software. In addition, its second purpose is to provide cost estimates for the work that will be involved in this integration project.

Visual Software Component Overview

Visual Software components will interface with instances of Management Information Systems at schools and with Dell supplied components within the data center. In general, our components will be connected as follows:

Overview – Click image to show in full screen

Staged Implementation

It has been indicated that, for the two implementations there may or may not be a Proof of Concept stage and that the two implementations themselves not include the exact same SIF coverage.  To make what we offer as flexible as possible, we shall split the UK SIF specification into a set of object groups. For each of the combinations of (implementation site and {POC and Production}), we will make recommendations as which groups would reasonably fall under the combination.

So, in the case of this document we will highlight our recommendation for the Cambridgeshire Proof of Concept, while leaving grayed out things that could be added as optional elements. We will document other combinations separately.

SIF Object Groups

The UK SIF specification does not group objects into clusters of any sort, but for the purposes of these documents, we will group them as follows:

• Basic: those objects needed for identity & role management, including learners, teachers and parents. It includes the schools that people with which people are associated, but not the school groups.
• Learner Groups: this extends the basic set to extend learner and teacher information to include learner group information. It might be argued that this group should be done at the same time timetabling, but a great deal of useful information can be derived from only these objects without the timetabling objects.
• Assessments: a group of interdependent objects that store information about assessments
• Attendance: these relay attendance information, either in summary or detailed information – in the UK, it is typical to send complete information (not just absences), so expect to receive a high volume of information
• Timetabling: this is information relative to how the school groups were established
• Uncategorized: these objects don’t fall into any of the above categories, but do include some important messages such as behavior, meal status and special education objects.

The following table shows which SIF objects are in each of these categories:

Recommendation for Cambridgeshire Production

It appears that almost all information covered in the current version of the SIF specification is required by the data warehouse specification distributed on June 28th, 2010, with the timetabling information not certain at the time of the writing of this document.

Connecting MIS Systems

The following diagram gives an overview of how the MIS systems will be connected into the SIF infrastructure:

On the Zone Integration Server, a separate zone will be created for each school (for more information on the use of Zones, see SIF Zones) and each school’s copy of Mimic will be set as the Provider of learner information in that zone. Mimic will generate SIF messages by looking at the CSV files that each copy of SIMS creates on a scheduled basis, comparing it against the copy it looked at on its last scheduled run. (For more information on how Mimic works, see Mimic Users Guide)

NOTE: A single ZIServer can host many (even thousands of) zones, and these zones can either be local (HTTP) or remote (HTTPS).

In order for the Mimic software to run, there will need to be a Windows computer running it and storing its SQL Server database in each location where there is an MIS.

Mimic as a Web Site

Like all SIF Agents, Mimic runs as a web site. Like many SIF Agents, Mimic may run in SIF Pull or Push mode (Pull mode is analogous to batch mode and Push mode is similar to real-time mode – the meanings are very different than the typical meanings of ‘Push’ and ‘Pull’ in English. For a more complete description, see SIF Push and Pull Modes.)

This has implications for the installation because SIF Push mode (real-time) will require that the installation at the school have an external IP number and that the ZIS be able to communicate with the Agent. Although this is more work, we recommend this configuration because it allows the school to have its own certificate and allows us to set up Encryption Level 4, Authentication Level 3 where each side verifies the certificate of the other.

It is not necessary to use an outside certificate authority (you could use Certificate Services) because we will be setting up a closed community of servers. NOTE: In E2BN, we found out that, although it is supposed to be a simple matter to get certificates whenever we needed them, in reality it wasn’t all that simple.

Mimic Database

We typically use SQL Server Express Edition for Mimic installations. We are recommending installing SQL Server 2008 R2, especially since it now expands to 10GB and allows users to back up and restore.

In this estimate, the length of time it would take to create and test the installation package will not change significantly with the differing number of SIF objects implemented; most of the time in doing that will be in the setup, testing and making sure that the correct results were achieved when the tests were run. The configuration of the Mimic product, however, will largely depend on the number of objects being configured. We assume here that even though one install package would have been created during the POC, another would be created at the end of the production effort.

Managing School Data Quality

Normally, applications have user interfaces that prevent things from automatically happening when it receives “bad” information. For example, an application that creates user accounts from information entered into a user interface (a form) would first validate that information before using it to create the account.

When a user interface is replaced with a SIF interface, the subscribing application might be left trusting the validation rules of the source system unless something is put between the two to make sure that data quality is being maintained.

For example, in an MIS, it might not be too important that learner’s birth date is reasonable – to the MIS, it is only important that a value is entered into the field and that it is a valid date. But, to a VLE it is important that it is valid as well as reasonable. If a user were entering this information by hand into the VLE, he or she would never be able to enter today’s date into the birth date field.

Veracity

Veracity is a product that does several things:

• It monitors data quality based on rules that have been set up at a certain level – for example, rules can be set up at the school level to capture errors as they come out of the MIS that will become problems in other systems that are connected to this system – checks like invalid birth dates, missing names, etc.
• It maintains a consolidated database of data collected from all systems that can be used for reporting or other data analysis
• It provides a portal interface so that users can see where they need to make changes to their data

Where Veracity Fits In

We recommend installing Veracity in two places in this architecture:

1. In the school zone – used here, the rule set that is applied to the collected data will check it against the needs of the other systems that will use its data. Additionally, the portal interface it provides will give users an insight into how close they are to the place where the data they enter can transparently flow throughout the system without alteration.
2. Post-Envoy – Envoy is the software that will match records and consolidate the information from the single zones into a group zone. Having Veracity subscribe here will provide a consolidated reporting and analysis database and a place where business rules can run that check for problems that might occur where data from one school collides with that from another.

The following diagram shows Veracity as it would be installed to monitor data quality for each of the school zones:

Adding Veracity – Click image to show in full screen

Since the ZIServer is being hosted, it is not unreasonable to assume that these copies of Veracity would be hosted as well.

Merging/Matching Data

The primary function of the Envoy product in this environment is to resolve differences when a physical object is  represented in more than one MIS. This can happen when a learner attends more than one school, a teacher teaches at more than one school or a parent has children attending more than one school.

When this occurs, what envoy does is to create a “virtual object” by choosing the base object from the “more reliable” of the two and making it appear as if there was a single large MIS providing all the records from a single source. What this accomplishes for subscribing applications is to provide them with a simplified view of the learner population and centralizing the complex, matching logic where it can be independently tested and verified.

For more information on how this works, see: Managed Virtual Zones.

In our implementation, we will use the Envoy product to perform this function, match records and combine records for all the schools into a single zone as depicted by the following diagram:

Adding Envoy – Click image to show in full screen

Setting up Envoy is more difficult than the other products because, for each of the SIF objects that it handles there are rules that must be, at a minimum, reviewed by the customer to make sure that everyone is in agreement as to how the object is being handled. These rules fall into two categories:

1. How records are matched. Envoy can use a hierarchy of rules when matching – for example, when matching learner records, it can first try ULN, family name, given name, post code and birth date. Then if that doesn’t match it can try ULN, family name, given name, gender and birth date, then three more combinations before giving up and assuming that the two records are for different people.
2. How the “more reliable” base object is chosen when there is a choice. If a child attends two schools, there are two LearnerPersonal records – which one should a subscribing application receive as the “more reliable” of the two? Sometimes it is easy to choose (because the SIF specification has rules for choosing, other times is is more difficult).

With Envoy, most of the more difficult review work will be in the Group 1 objects. This is because Group 1 includes all of the “base” objects that are the ones that get matched: LAInfo, SchoolInfo, LearnerPersonal, ContactPersonal and WorkforcePersonal. These objects will have the hierarchy of matching rules that need to be reviewed where the other objects will have simpler, more definitive rules (however, they will typically have many more of them). This is why the bulk of the hours show up in the estimate for Group 1.

NOTE: At this point, it would be possible to connect a “Local Authority Level” SIF-enabled application to the consolidated zone (as we intend to do with Ipseity).

Identity and Role Management

Once records have been data quality checked and consolidated, information is passed to the Identity and Role Manager, Ipseity™ (formerly codename Sentry). This SIF agent will subscribe to several objects and publish one – by subscribing, it will receive all of the information it needs to create account names and most of the information it needs to assign roles to the applications it manages.

The following diagram gives an overall picture of where Ipseity fits into the overall architecture:

 Adding Ipseity – Click image to show in full screen

As shown in this picture, Ipseity has not only a SIF interface, but also a web services interface. This interface was put in so that non-SIF applications could easily use it features such as:

• Adding new users to the pool of users being managed, so that Ipseity can assign IDs that are consistent with the automatically managed IDs
• Query the database of all users for {new/changed/deleted} users, which users are assigned to which schools, which users are assigned to what roles (in which schools), etc..

There are several web service calls supported by Ipseity that all return recordsets, making it simple for other applications to access data in its store either from the same location or remotely (through HTTPS). For more information on these facilities, see: Extending Identity Management – Roles

Subscribed Objects

Ipseity will subscribe to the following objects:

• LAInfo
• SchoolInfo
• LearnerPersonal
• LearnerSchoolEnrolment
• WorkforcePersonal – this contains all information about teachers, including all schools at which they work. Some of these records may be skipped because their assignments might indicate that they have no computer interaction
• ContactPersonal – this contains basic information about a parent or other contact. Some of these records will be ignored, such as those for after school programs or medical facilities that are stored in an MIS, but will not be needed in Ipseity for Identity or Role management purposes
• LearnerContact – this links a contact record to a learner record

Published Object

Ipseity will publish the following object:

• Identity – this will contain the identities it assigns, including AD account names, Shibboleth names, and initial passwords, etc.

Roles Being Managed

At a minimum, roles for the products we are installing will need to be managed.

For example, ZIServer will need to be assigned people who will be able to create and manage zones, assign permissions and those who will have access to the audits. It might be the case that people in the schools may need access to the audits as well.

Veracity’s interface will need a similar administrator role, but it will also have interfaces for a person who creates business rules (someone who knows about school MIS data – probably not a system administrator), and several people in each school who will be using its interface to correct errors in data (these will probably not overlap with the other two at all).

Envoy will need an administrator, but will also need someone who oversees the business rules that govern how record matching is done and choices are made (this person is more of an analyst, not a systems administrator), but there will be nobody in the school who will have access.

Question: In the following spreadsheet, it was indicated that the following roles are required for use by one or more Dell applications:

Original Excel file with edits:Sims MIS role data fields with SIF code values.xls

Would it be desirable to use Ipseity to manage these roles along with the roles for ZIServer, Veracity and Envoy? Users would be able to use the same interface and the applications would have access to the common web services interface.

Note: the setup of this application does not depend on the SIF objects being configured, so it will not vary from implementation to implementation.

Consolidated Data Quality Checking / Reporting

The final step in the process (or this could be swapped with the IDRM step) will be to set up a copy of Veracity that consolidates all data from all schools (and Ipseity) into a single database and performs another set of data quality rules on them.

The data quality rules that are run on this set of data should not be checking for the simple things that should have been remedied at the school, but rather other types of errors that show up when data from one school is combined with that from another.

An example might be: for a given learner we might see attendance records showing him attending both his morning and afternoon schools the same morning of the same day and this is not supposed to happen. Or, we see two children with completely different demographic information in different schools, but they have the same ULN.

The following diagram shows where this copy of Veracity is installed in the architecture:

 Adding Veracity (on consolidated data set) – Click image to show in full screen

This copy becomes a subscriber in the combined zone. Since Veracity’s database is effectively an Enterprise Data Store, we’ve found that it is very useful to other applications such as:

• Data Warehouses: these can sample the database on given intervals. Typically, the DW builds a series of views, stored procedures or compiled queries that get called on a regular basis that take a snapshot from our repository.
• Reporting Application: tools such as SQL Reporting Services are used in districts to produced real-time reports on data that is collected from many sources. In our scenario, we are only collecting data from the MIS and from Ipseity, but in other places they may collect data from as many as 8 or 9 places.
• Data Analysis: some use SQL Analysis in similar ways to perform data analysis on the Veracity database or a replicated snapshot.

To view the database layout and to get an empty Veracity Database Backup, see: Veracity UK 1.2 Data

Summary

This is an estimate; these hours are based on certain assumptions about a number of business rules that will be requested and a level of data cleanliness we will find when we see the actual data.

This could vary widely – the difference in the effort required for us to successfully integrate all these applications between locations where the data is carefully maintained and those where it isn’t can be huge (up to 50% of the project).  This is why we suggest a step-by-step approach where the data quality issues are addressed in the first step, so that they will not haunt us in later steps.

We also use the strategy of giving tools to users in the first step that “make their life better”. One would think that Veracity “looking over their shoulder” would not fit into that category, but it can.

For example, if we were to encode their annual census rules into quality checks, it would give them a tool that allows them to see how close they are to not having any work to do next time a census is due – except to run a query against their local Veracity DB and export it – perhaps…

These estimates assume that the work would be able to be performed from our office through a connection such as a VPN tunnel. We found that working offsite (such as we did during the first POC) added about 80% to the length of time it took to do a given task (so 10 hours in our office working through a remote facility = 18 hours working at the Dell site).

This SOW is intended to be a working document – the comments facility is here to allow other team members to pose questions, make comments and give suggestions – please use it. Comments will first go through me (Rob Hutchison) for approval, so if you don’t want it to be seen by everyone, put “PRIVATE” on the first line and I won’t publish it for all. Thanks.

Introduction

In June 2009, Dell, Visual Software and ITWorx assembled a Proof of Concept for the Cambridgeshire BSF bid and demonstrated that the Schools Interoperability Framework could be used to transport learner information such as that which would come from a typical UK MIS to a learning platform in near real time to meet the requirements of the Local Authority (LA).

In the first phase, for several reasons, we used a MIS simulator instead of using a copy of Capita SIMS, Serco CMIS or one of the other MIS systems.  In the second phase of this overall project, we will be putting together a lasting platform that will be used in the Cambridgeshire and Derbyshire implementations.

This document’s first purpose is to supply Visual Software’s recommendations for setting up a SIF implementation for these schools, connecting the MIS instances, providing matching, identity and role management, providing facilities for checking the data for quality standards and providing an interface for the other Dell-incorporated software. In addition, its second purpose is to provide cost estimates for the work that will be involved in this integration project.

Visual Software Component Overview

Visual Software components will interface with instances of Management Information Systems at schools and with Dell supplied components within the data center. In general, our components will be connected as follows:

Overview – Click image to show in full screen

Staged Implementation

It has been indicated that, for the two implementations there may or may not be a Proof of Concept stage and that the two implementations themselves not include the exact same SIF coverage.  To make what we offer as flexible as possible, we shall split the UK SIF specification into a set of object groups. For each of the combinations of (implementation site and {POC and Production}), we will make recommendations as which groups would reasonably fall under the combination.

So, in the case of this document we will highlight our recommendation for the Cambridgeshire Proof of Concept, while leaving grayed out things that could be added as optional elements. We will document other combinations separately.

SIF Object Groups

The UK SIF specification does not group objects into clusters of any sort, but for the purposes of these documents, we will group them as follows:

• Basic: those objects needed for identity & role management, including learners, teachers and parents. It includes the schools that people with which people are associated, but not the school groups.
• Learner Groups: this extends the basic set to extend learner and teacher information to include learner group information. It might be argued that this group should be done at the same time timetabling, but a great deal of useful information can be derived from only these objects without the timetabling objects.
• Assessments: a group of interdependent objects that store information about assessments
• Attendance: these relay attendance information, either in summary or detailed information – in the UK, it is typical to send complete information (not just absences), so expect to receive a high volume of information
• Timetabling: this is information relative to how the school groups were established
• Uncategorized: these objects don’t fall into any of the above categories, but do include some important messages such as behavior, meal status and special education objects.

The following table shows which SIF objects are in each of these categories:

Recommendation for Cambridgeshire POC

Since the goal of this POC is to implement Identity and Role management and to supply Forefront Identity Manager with the basic information it needs to create and manage accounts, we recommend that we implement “Group 1: Basic” SIF objects in this phase.

Connecting MIS Systems

The following diagram gives an overview of how the MIS systems will be connected into the SIF infrastructure:

On the Zone Integration Server, a separate zone will be created for each school (for more information on the use of Zones, see SIF Zones) and each school’s copy of Mimic will be set as the Provider of learner information in that zone. Mimic will generate SIF messages by looking at the CSV files that each copy of SIMS creates on a scheduled basis, comparing it against the copy it looked at on its last scheduled run. (For more information on how Mimic works, see Mimic Users Guide)

NOTE: A single ZIServer can host many (even thousands of) zones, and these zones can either be local (HTTP) or remote (HTTPS).

In order for the Mimic software to run, there will need to be a Windows computer running it and storing its SQL Server database in each location where there is an MIS.

Mimic as a Web Site

Like all SIF Agents, Mimic runs as a web site. Like many SIF Agents, Mimic may run in SIF Pull or Push mode (Pull mode is analogous to batch mode and Push mode is similar to real-time mode – the meanings are very different than the typical meanings of ‘Push’ and ‘Pull’ in English. For a more complete description, see SIF Push and Pull Modes.)

This has implications for the installation because SIF Push mode (real-time) will require that the installation at the school have an external IP number and that the ZIS be able to communicate with the Agent. Although this is more work, we recommend this configuration because it allows the school to have its own certificate and allows us to set up Encryption Level 4, Authentication Level 3 where each side verifies the certificate of the other.

It is not necessary to use an outside certificate authority (you could use Certificate Services) because we will be setting up a closed community of servers. NOTE: In E2BN, we found out that, although it is supposed to be a simple matter to get certificates whenever we needed them, in reality it wasn’t all that simple.

Mimic Database

We typically use SQL Server Express Edition for Mimic installations. We are recommending installing SQL Server 2008 R2, especially since it now expands to 10GB and allows users to back up and restore.

In this estimate, the length of time it would take to create and test the installation package will not change significantly with the differing number of SIF objects implemented; most of the time in doing that will be in the setup, testing and making sure that the correct results were achieved when the tests were run. The configuration of the Mimic product, however, will largely depend on the number of objects being configured.

Managing School Data Quality

Normally, applications have user interfaces that prevent things from automatically happening when it receives “bad” information. For example, an application that creates user accounts from information entered into a user interface (a form) would first validate that information before using it to create the account.

When a user interface is replaced with a SIF interface, the subscribing application might be left trusting the validation rules of the source system unless something is put between the two to make sure that data quality is being maintained.

For example, in an MIS, it might not be too important that learner’s birth date is reasonable – to the MIS, it is only important that a value is entered into the field and that it is a valid date. But, to a VLE it is important that it is valid as well as reasonable. If a user were entering this information by hand into the VLE, he or she would never be able to enter today’s date into the birth date field.

Veracity

Veracity is a product that does several things:

• It monitors data quality based on rules that have been set up at a certain level – for example, rules can be set up at the school level to capture errors as they come out of the MIS that will become problems in other systems that are connected to this system – checks like invalid birth dates, missing names, etc.
• It maintains a consolidated database of data collected from all systems that can be used for reporting or other data analysis
• It provides a portal interface so that users can see where they need to make changes to their data

Where Veracity Fits In

We recommend installing Veracity in two places in this architecture:

1. In the school zone – used here, the rule set that is applied to the collected data will check it against the needs of the other systems that will use its data. Additionally, the portal interface it provides will give users an insight into how close they are to the place where the data they enter can transparently flow throughout the system without alteration.
2. Post-Envoy – Envoy is the software that will match records and consolidate the information from the single zones into a group zone. Having Veracity subscribe here will provide a consolidated reporting and analysis database and a place where business rules can run that check for problems that might occur where data from one school collides with that from another.

The following diagram shows Veracity as it would be installed to monitor data quality for each of the school zones:

Adding Veracity – Click image to show in full screen

Since the ZIServer is being hosted, it is not unreasonable to assume that these copies of Veracity would be hosted as well.

Merging/Matching Data

The primary function of the Envoy product in this environment is to resolve differences when a physical object is  represented in more than one MIS. This can happen when a learner attends more than one school, a teacher teaches at more than one school or a parent has children attending more than one school.

When this occurs, what envoy does is to create a “virtual object” by choosing the base object from the “more reliable” of the two and making it appear as if there was a single large MIS providing all the records from a single source. What this accomplishes for subscribing applications is to provide them with a simplified view of the learner population and centralizing the complex, matching logic where it can be independently tested and verified.

For more information on how this works, see: Managed Virtual Zones.

In our implementation, we will use the Envoy product to perform this function, match records and combine records for all the schools into a single zone as depicted by the following diagram:

Adding Envoy – Click image to show in full screen

Setting up Envoy is more difficult than the other products because, for each of the SIF objects that it handles there are rules that must be, at a minimum, reviewed by the customer to make sure that everyone is in agreement as to how the object is being handled. These rules fall into two categories:

1. How records are matched. Envoy can use a hierarchy of rules when matching – for example, when matching learner records, it can first try ULN, family name, given name, post code and birth date. Then if that doesn’t match it can try ULN, family name, given name, gender and birth date, then three more combinations before giving up and assuming that the two records are for different people.
2. How the “more reliable” base object is chosen when there is a choice. If a child attends two schools, there are two LearnerPersonal records – which one should a subscribing application receive as the “more reliable” of the two? Sometimes it is easy to choose (because the SIF specification has rules for choosing, other times is is more difficult).

With Envoy, most of the more difficult review work will be in the Group 1 objects. This is because Group 1 includes all of the “base” objects that are the ones that get matched: LAInfo, SchoolInfo, LearnerPersonal, ContactPersonal and WorkforcePersonal. These objects will have the hierarchy of matching rules that need to be reviewed where the other objects will have simpler, more definitive rules (however, they will typically have many more of them). This is why the bulk of the hours show up in the estimate for Group 1.

NOTE: At this point, it would be possible to connect a “Local Authority Level” SIF-enabled application to the consolidated zone (as we intend to do with Ipseity).

Identity and Role Management

Once records have been data quality checked and consolidated, information is passed to the Identity and Role Manager, Ipseity™ (formerly codename Sentry). This SIF agent will subscribe to several objects and publish one – by subscribing, it will receive all of the information it needs to create account names and most of the information it needs to assign roles to the applications it manages.

The following diagram gives an overall picture of where Ipseity fits into the overall architecture:

 Adding Ipseity – Click image to show in full screen

As shown in this picture, Ipseity has not only a SIF interface, but also a web services interface. This interface was put in so that non-SIF applications could easily use it features such as:

• Adding new users to the pool of users being managed, so that Ipseity can assign IDs that are consistent with the automatically managed IDs
• Query the database of all users for {new/changed/deleted} users, which users are assigned to which schools, which users are assigned to what roles (in which schools), etc..

There are several web service calls supported by Ipseity that all return recordsets, making it simple for other applications to access data in its store either from the same location or remotely (through HTTPS). For more information on these facilities, see: Extending Identity Management – Roles

Subscribed Objects

Ipseity will subscribe to the following objects:

• LAInfo
• SchoolInfo
• LearnerPersonal
• LearnerSchoolEnrolment
• WorkforcePersonal – this contains all information about teachers, including all schools at which they work. Some of these records may be skipped because their assignments might indicate that they have no computer interaction
• ContactPersonal – this contains basic information about a parent or other contact. Some of these records will be ignored, such as those for after school programs or medical facilities that are stored in an MIS, but will not be needed in Ipseity for Identity or Role management purposes
• LearnerContact – this links a contact record to a learner record

Published Object

Ipseity will publish the following object:

• Identity – this will contain the identities it assigns, including AD account names, Shibboleth names, and initial passwords, etc.

Roles Being Managed

At a minimum, roles for the products we are installing will need to be managed.

For example, ZIServer will need to be assigned people who will be able to create and manage zones, assign permissions and those who will have access to the audits. It might be the case that people in the schools may need access to the audits as well.

Veracity’s interface will need a similar administrator role, but it will also have interfaces for a person who creates business rules (someone who knows about school MIS data – probably not a system administrator), and several people in each school who will be using its interface to correct errors in data (these will probably not overlap with the other two at all).

Envoy will need an administrator, but will also need someone who oversees the business rules that govern how record matching is done and choices are made (this person is more of an analyst, not a systems administrator), but there will be nobody in the school who will have access.

Question: In the following spreadsheet, it was indicated that the following roles are required for use by one or more Dell applications:

Original Excel file with edits:Sims MIS role data fields with SIF code values.xls

Would it be desirable to use Ipseity to manage these roles along with the roles for ZIServer, Veracity and Envoy? Users would be able to use the same interface and the applications would have access to the common web services interface.

Note: the setup of this application does not depend on the SIF objects being configured, so it will not vary from implementation to implementation.

Consolidated Data Quality Checking / Reporting

The final step in the process (or this could be swapped with the IDRM step) will be to set up a copy of Veracity that consolidates all data from all schools (and Ipseity) into a single database and performs another set of data quality rules on them.

The data quality rules that are run on this set of data should not be checking for the simple things that should have been remedied at the school, but rather other types of errors that show up when data from one school is combined with that from another.

An example might be: for a given learner we might see attendance records showing him attending both his morning and afternoon schools the same morning of the same day and this is not supposed to happen. Or, we see two children with completely different demographic information in different schools, but they have the same ULN.

The following diagram shows where this copy of Veracity is installed in the architecture:

 Adding Veracity (on consolidated data set) – Click image to show in full screen

This copy becomes a subscriber in the combined zone. Since Veracity’s database is effectively an Enterprise Data Store, we’ve found that it is very useful to other applications such as:

• Data Warehouses: these can sample the database on given intervals. Typically, the DW builds a series of views, stored procedures or compiled queries that get called on a regular basis that take a snapshot from our repository.
• Reporting Application: tools such as SQL Reporting Services are used in districts to produced real-time reports on data that is collected from many sources. In our scenario, we are only collecting data from the MIS and from Ipseity, but in other places they may collect data from as many as 8 or 9 places.
• Data Analysis: some use SQL Analysis in similar ways to perform data analysis on the Veracity database or a replicated snapshot.

To view the database layout and to get an empty Veracity Database Backup, see: Veracity UK 1.2 Data

Data Warehouse Integration

We would assume that if the Veracity database is to be integrated with an external database that we would be called on for support during the process. Based on interactions we’ve had with partners we have worked with before who have done similar activities, we would recommend addition an additional 30 hours developer time to the estimate to accommodate this activity.

Summary

This is an estimate; these hours are based on certain assumptions about a number of business rules that will be requested and a level of data cleanliness we will find when we see the actual data.

This could vary widely – the difference in the effort required for us to successfully integrate all these applications between locations where the data is carefully maintained and those where it isn’t can be huge (up to 50% of the project).  This is why we suggest a step-by-step approach where the data quality issues are addressed in the first step, so that they will not haunt us in later steps.

We also use the strategy of giving tools to users in the first step that “make their life better”. One would think that Veracity “looking over their shoulder” would not fit into that category, but it can.

For example, if we were to encode their annual census rules into quality checks, it would give them a tool that allows them to see how close they are to not having any work to do next time a census is due – except to run a query against their local Veracity DB and export it – perhaps…

These estimates assume that the work would be able to be performed from our office through a connection such as a VPN tunnel. We found that working offsite (such as we did during the first POC) added about 80% to the length of time it took to do a given task (so 10 hours in our office working through a remote facility = 18 hours working at the Dell site).

This SOW is intended to be a working document – the comments facility is here to allow other team members to pose questions, make comments and give suggestions – please use it. Comments will first go through me (Rob Hutchison) for approval, so if you don’t want it to be seen by everyone, put “PRIVATE” on the first line and I won’t publish it for all. Thanks.

Web Service Method LEGEND
Click on the name of a method to view that method’s description.
Method  Names:	Method Name
	AddScope	AddUser	AssignUserToScope	AssignUserWithinScope ToPermission
	ChangeScope	ChangeUser	ChangeUserStatus	DeleteScope
	DeleteUser	FindRoleAndScope	FindRoleAndScopeAndApp	FindUsersAndRoles AndApps
	GetRecentUserInfo	GetScopeInfo	GetUserInfo	RemoveUserFromScope

The Active Directory Web Service is the component of ZIAgent AD which is responsible for directly interacting with Active Directory. This document explains each of the methods of this web-service in detail.

RenameAccount

Purpose

Used to change the name of an pre-existing account. This is most commonly used to accomodate situations where account names are based on a user’s first and/or last name.

Parameters

• OldCN: The container name of the existing account. Example: Michael.Shea
• NewCN: The desired container name. Example: Mike.Shea
• NewDisplayName: The desired display name. Example: Shea, Mike

Actions Taken

• Sets CN to new value. Example: Changes Michael.Shea to Mike.Shea
• Sets property userPrincipalName. Example: Mike.Shea@visualsi.com
• Sets property sAMAccountName. Example: Changes Michael.Shea to Mike.Shea
• Sets property displayName.

Return Value

• On success: Returns "SUCCESS:IDENTITY:LDAP://…."
• On fail: Returns "FAILURE:…

SyncAccount

Purpose

Used to create the desired account if it does not already exist, or, if the account already exists, move to the proper location (OU).

Parameters

• OULDAP: The complete LDAP of the OU where the account should be created or moved to. Example: LDAP://OU=Developers,OU=Employees,DC=visualsi,DC=com
• CN: The container name of the account to be created or moved. Example: Mike.Shea
• DisplayName: The desired display name.
• MustChangePassword: Indicates if this user will be forced to change their password on next login. Example: TRUE
• PasswordNeverExpires: Indicates if this user’s password never expires. Example: FALSE
• Password: The desired password if this is a new account. This value must conform to the password policy in effect on the domain controller. Example: MyPassw0rd

Actions Taken

• If the specified account already exists in the specified OU, no action is taken.
• If the specified account already exists somewhere other than the specified OU, it will be moved to the proper OU.
• If the specified account does not exist, it will be created in the specified OU.
  • Sets CN to specified value.
  • Sets property userPrincipalName. Example: Mike.Shea@visualsi.com
  • Sets property sAMAccountName. Example: Changes Michael.Shea to Mike.Shea
  • Sets property displayName. Example: Shea, Mike
  • Sets the password on the new account.
  • Disables the account.
  • Sets the Password Never Expires flag on the account.
  • Sets the User Must Change Password flag on the account.

Return Value

• On success: Returns "SUCCESS:IDENTITY:LDAP://…."
• On fail: Returns "FAILURE:…

SyncAccountHomeDir

Purpose

To create, update, or move a home directory.

Parameters

• AccountLDAP: The complete LDAP of the account whose home directory is to be effected. Example: LDAP://CN=Mike.Shea,OU=Developers,OU=Employees,DC=visualsi,DC=com
• HomeDirTemplatePath: The path to the directory where any default files/folders exist which need to be added to this home directory. Example: MyServerHomeDirectoryTemplatesEmployee
• HomeDirPath: The path to the directory where the home directory should be located. Example: MyServerHomeDirectoriesEmployees
• HomeDirName: The name of the home directory. Example: MikeShea
• HomeDrive: The mapped drive letter to the home directory. Example: H:
• MoveHomeDir: Specifies if the home directory should be moved to the specified location. Example: TRUE
• ResetPermissionsOnExistingDir: Specifies if permissions should be reset on the home directory if it already exists. Example: FALSE

Actions Taken

• Checks to verify the specified account exists. If it does not, no further action is taken.
• If the specified account exists, retrieves the current homeDirectory property.
• If the homeDirectory property already had a value, and MoveHomeDir is set to TRUE, then attempt to copy the existing home directory (including its contents) to the new location. The original home directory and its contents are left intact in their original location.
• If the homeDirectory property already had a value, and MoveHomeDir is set to FALSE, then create the home directory in the location specified by the homeDirectory property if the home directory does not already exist there.
• If the homeDirectory property did not have a value, then create the home directory in the specified location if it does not already exist. Any files/folders in the HomeDirTemplatePath will be copied into the home directory if it was just created.

Return Value

• On success: Returns "SUCCESS:IDENTITY:MyServerHomeDirectoriesEmployeesMikeShea"
• On fail: Returns "FAILURE:…

UpdateAccount

Purpose

To update a set of pre-defined properties of an existing account.

Parameters

• AccountLDAP: The complete LDAP of the account to update. Example: LDAP://CN=Mike.Shea,OU=Developers,OU=Employees,DC=visualsi,DC=com
• EnableAccount: Enables/disables the account. Example: TRUE
• Description: Sets the Description property.
• FirstName: Sets the givenName property. Example: Mike
• LastName: Sets the sn property. Example: Shea
• MiddleInitial: Sets the initials property. Example: C
• Email: Sets the mail property. Example: mike.shea@visualsi.com
• HomePhone: Sets the homephone property. Example: 215.493.8210
• MobilePhone: Sets the mobile property. Example: 215.493.8210
• StreetAddress: Sets the streetAddress property. Example: 1105 Taylorsville Road
• POBox: Sets the postOfficeBox property. Example: 398
• City: Sets the l property. Example: Washington Crossing
• State: Sets the st property. Example: NJ
• PostalCode: Sets the postalCode property. Example: 18977
• EmployeeID: Sets the employeeID property. Example: 1234
• Company: Sets the company property. Example: Visual Software
• Department: Sets the department property. Example: Development
• Title: Sets the title property. Example: Developer
• OfficePhone: Sets the telephoneNumber property. Example: 215.493.8210
• Manager: Sets the manager property. Must be the CN of this user’s manager’s account. Example: Robert.Hutchison

Actions Taken

• Sets each property described in Parameters to the specified value.
• To ignore a property and not set its value to anything, pass in the value "NONE" for that property.
• To clear a property’s value, pass in an empty string for that property.

Return Value

• On success: Returns "SUCCESS:IDENTITY:LDAP://…."
• On fail: Returns "FAILURE:…

UpdateAccountProperty

Purpose

To update a specific property on an existing account.

Parameters

• AccountLDAP: The complete LDAP of the account to update. Example: LDAP://CN=Mike.Shea,OU=Developers,OU=Employees,DC=visualsi,DC=com
• PropertyName: The name of the property to update. Example: mail
• PropertyValue: The value to assign to the specified property. Example: mike.shea@visualsi.com

Actions Taken

• Assigns the specified value to the specified property.
• To ignore a property and not set its value to anything, pass in the value "NONE" for that property.
• To clear a property’s value, pass in an empty string for that property.
• In addition to the standard list of active directory properties, the following "special" values may be passed in using the PropertyName parameter:
  • AccountAvailability: Enables/disables an account. Acceptable values are "TRUE" and "FALSE".
  • SetPassword: Assigns a new password to the account. The value must meet the domain password policy.
  • MustChangePassword: Determines if the user must change their password on next login. Acceptable values are "TRUE" and "FALSE".
  • PasswordExpires: Determines if the account password expires. Acceptable values are "TRUE" and "FALSE".
  • SetManager: Associates a manager’s ad account with this account. Value must be a valid CN.
  • UserCannotChangePwd: Determines if the user cannot change their password. Acceptable values are "TRUE" and "FALSE".

Return Value

• On success: Returns "SUCCESS"
• On fail: Returns "FAILURE:…

SyncAccountToGroup

Purpose

To assign a pre-existing account to a pre-existing security group.

Parameters

• AccountLDAP: The complete LDAP of the account to update. Example: LDAP://CN=Mike.Shea,OU=Developers,OU=Employees,DC=visualsi,DC=com
• GroupLDAP: The complete LDAP of the security group to add the account to. Example: LDAP://CN=Developers,OU=Employees,DC=visualsi,DC=com

Actions Taken

• If the account or security group do not exist, no action is taken.
• If the account and the security group exist and the account is not already a member of the group, the account is added to the group.

Return Value

• On success: Returns "SUCCESS:IDENTITY:LDAP://…."
• On fail: Returns "FAILURE:…

RemoveAccountFromGroups

Purpose

To remove an account from one or more security groups.

Parameters

• AccountLDAP: The complete LDAP of the account to update. Example: LDAP://CN=Mike.Shea,OU=Developers,OU=Employees,DC=visualsi,DC=com
• GroupLDAP: The complete LDAP of the security group to remove the account from. An empty string is also an acceptable value. Example: LDAP://CN=Developers,OU=Employees,DC=visualsi,DC=com

Actions Taken

• If a GroupLDAP is not specified, the account will be removed from all of its current security groups.
• If a GroupLDAP is specified, the account will be removed from only that security group.

Return Value

• On success: Returns "SUCCESS"
• On fail: Returns "FAILURE:…

SyncGroupToHomeDir

Purpose

To allow a security group permission to access a home directory.

Parameters

• GroupLDAP: The complete LDAP of the security group. Example: LDAP://CN=Developers,OU=Employees,DC=visualsi,DC=com
• HomeDirPath: The path to the home directory. Example: MyServerHomeDirectoriesEmployeesMikeShea

Actions Taken

• If the group and home directory do not exist no action is taken.
• If the group and home directory exist, the group is assigned permission to access the home directory.

Return Value

• On success: Returns "SUCCESS"
• On fail: Returns "FAILURE:…

RemoveGroupFromHomeDir

Purpose

To remove a security group’s permission from a home directory.

Parameters

• GroupLDAP: The complete LDAP of the security group. Example: LDAP://CN=Developers,OU=Employees,DC=visualsi,DC=com
• HomeDirPath: The path to the home directory. Example: MyServerHomeDirectoriesEmployeesMikeShea

Actions Taken

• If the group and home directory do not exist no action is taken.
• If the group does not currently have permission on the home directory no action is taken.
• If the group and home directory exist, and the group has permission on the home directory, the group is removed from the home directory.

Return Value

• On success: Returns "SUCCESS"
• On fail: Returns "FAILURE:…

SyncOU

Purpose

To create or update an organizational unit (OU).

Parameters

• ParentLDAP: The complete LDAP of the OU which contains the OU being created or updated. Example: LDAP://OU=Employees,DC=visualsi,DC=com
• OU: The name of the oganizational unit to create or update. Example: Developers
• Description: The description of the organizational unit to create or update.

Actions Taken

• If the OU does not exist in the OU specified in ParentLDAP, it will be created.
• If the OU already exists in the specified ParentLDAP, its description will be updated.

Return Value

• On success: Returns "SUCCESS:IDENTITY:LDAP://…."
• On fail: Returns "FAILURE:…

SyncGroup

Purpose

To create or update a security group.

Parameters

• OULDAP: The complete LDAP of the OU in which this security group should exist. Example: LDAP://OU=Developers,OU=Employees,DC=visualsi,DC=com
• CN: The name of the security group to be created or updated. Example: Developers
• Description: The description of the security group.
• GroupType: The type of security group to create or update. Acceptable values are "GlobalDistribution", "GlobalSecurity", "UniversalDistribution", "UniversalSecurity". If no type is specified, "GlobalSecurity" is assumed.

Actions Taken

• If the group does not already exist in the specified OU, it will be created. The new group’s sAMAccountName property will be set to the CN, and the description property will be set to Description.
• If the group already exists in the specified OU, the description property is set.

Return Value

• On success: Returns "SUCCESS:IDENTITY:LDAP://…."
• On fail: Returns "FAILURE:…

DiscoverAccounts

Purpose

To discover information about pre-existing active directory accounts in order to syncronize them with SIF data and prevent duplicate accounts from being created.

Parameters

• ObjectName: The name of the SIF object that discovered accounts should be associated with. Example: StudentPersonal
• OULDAP: The LDAP in which to search active directory for accounts. Example: LDAP://OU=Employees,DC=visualsi,DC=com
• LocalIdProperty: The name of the active directory property which uniquely identifies accounts. This property should contain a value which exists in SIF. Example: employeeID
• SearchChildren: Indicates if the specified OU should be searched for accounts, or if this OU and all of its child OU’s should be searched. Example: TRUE

Actions Taken

• For each account found, a row is inserted to the table AD_DiscoveredAccounts in the ZIAgent database. The data inserted contains information identifying the account, including the SIF object name it is associated with, its LDAP, unique id, CN, OU, Home Directory, and Home Directory Drive.
• If SearchChildren is TRUE, searches the specified OU and all child OUs for accounts. Otherwise, searches only the specified OU.

Return Value

• On success: Returns "SUCCESS"
• On fail: Returns "FAILURE:…

Introduction

In June 2009, Dell, Visual Software and ITWorx assembled a Proof of Concept for the Cambridgeshire BSF bid and demonstrated that the Schools Interoperability Framework could be used to transport learner information such as that which would come from a typical UK MIS to a learning platform in near real time to meet the requirements of the Local Authority (LA).

In the first phase, for several reasons, we used a MIS simulator instead of using a copy of Capita SIMS, Serco CMIS or one of the other MIS systems.  In the second phase of this overall project, we will be putting together a lasting platform that will be used in the Cambridgeshire and Derbyshire implementations.

This document’s first purpose is to supply Visual Software’s recommendations for setting up a SIF implementation for these schools, connecting the MIS instances, providing matching, identity and role management, providing facilities for checking the data for quality standards and providing an interface for the other Dell-incorporated software. In addition, its second purpose is to provide cost estimates for the work that will be involved in this integration project.

Visual Software Component Overview

Visual Software components will interface with instances of Management Information Systems at schools and with Dell supplied components within the data center. In general, our components will be connected as follows:

Overview – Click image to show in full screen

Staged Implementation

It has been indicated that, for the two implementations there may or may not be a Proof of Concept stage and that the two implementations themselves not include the exact same SIF coverage.  To make what we offer as flexible as possible, we shall split the UK SIF specification into a set of object groups. For each of the combinations of (implementation site and {POC and Production}), we will make recommendations as which groups would reasonably fall under the combination.

In the case of this document we will highlight our recommendation for the Derbyshire Production, assuming that a Proof of Concept preceded it, while leaving grayed out things that could be added as optional elements. We are also assuming that the production environment will be used to support:

• Identity and Role Management
• Veracity Data Quality Management
• Forefront Identity Management

If other applications, such as learning management systems are needed, they will likely add objects from other SIF object groups.

SIF Object Groups

The UK SIF specification does not group objects into clusters of any sort, but for the purposes of these documents, we will group them as follows:

• Basic: those objects needed for identity & role management, including learners, teachers and parents. It includes the schools that people with which people are associated, but not the school groups.
• Learner Groups: this extends the basic set to extend learner and teacher information to include learner group information. It might be argued that this group should be done at the same time as Timetabling, but a great deal of useful information can be derived from only these objects without the timetabling objects.
• Assessments: a group of interdependent objects that store information about assessments
• Attendance: these relay attendance information, either in summary or detailed information – in the UK, it is typical to send complete information (not just absences), so expect to receive a high volume of information
• Timetabling: this is information relative to how the school groups were established
• Uncategorized: these objects don’t fall into any of the above categories, but do include some important messages such as behavior, meal status and special education objects.

The following table shows which SIF objects are in each of these categories:

Recommendation for Derbyshire Production

The highlighted objects are those we believe will be needed to support Identity and Role management using Ipseity and Microsoft FIM products.

Connecting MIS Systems

The following diagram gives an overview of how the MIS systems will be connected into the SIF infrastructure:

On the Zone Integration Server, a separate zone will be created for each school (for more information on the use of Zones, see SIF Zones) and each school’s copy of Mimic will be set as the Provider of learner information in that zone. Mimic will generate SIF messages by looking at the CSV files that each copy of SIMS creates on a scheduled basis, comparing it against the copy it looked at on its last scheduled run. (For more information on how Mimic works, see Mimic Users Guide)

NOTE: A single ZIServer can host many (even thousands of) zones, and these zones can either be local (HTTP) or remote (HTTPS).

In order for the Mimic software to run, there will need to be a Windows computer running it and storing its SQL Server database in each location where there is an MIS.

Mimic as a Web Site

Like all SIF Agents, Mimic runs as a web site. Like many SIF Agents, Mimic may run in SIF Pull or Push mode (Pull mode is analogous to batch mode and Push mode is similar to real-time mode – the meanings are very different than the typical meanings of ‘Push’ and ‘Pull’ in English. For a more complete description, see SIF Push and Pull Modes.)

This has implications for the installation because SIF Push mode (real-time) will require that the installation at the school have an external IP number and that the ZIS be able to communicate with the Agent. Although this is more work, we recommend this configuration because it allows the school to have its own certificate and allows us to set up Encryption Level 4, Authentication Level 3 where each side verifies the certificate of the other.

It is not necessary to use an outside certificate authority (you could use Certificate Services) because we will be setting up a closed community of servers. NOTE: In E2BN, we found out that, although it is supposed to be a simple matter to get certificates whenever we needed them, in reality it wasn’t all that simple.

Mimic Database

We typically use SQL Server Express Edition for Mimic installations. We are recommending installing SQL Server 2008 R2, especially since it now expands to 10GB and allows users to back up and restore.

In this estimate, the length of time it would take to create and test the installation package will not change significantly with the differing number of SIF objects implemented; most of the time in doing that will be in the setup, testing and making sure that the correct results were achieved when the tests were run. The configuration of the Mimic product, however, will largely depend on the number of objects being configured.

*NOTE: In Group 6 items, only three objects are recommended for Derbyshire. Adding them should only add 9 additional hours to the configuration process and shouldn’t add any additional time to the time it takes to create an installation package.

Managing School Data Quality

Normally, applications have user interfaces that prevent things from automatically happening when it receives “bad” information. For example, an application that creates user accounts from information entered into a user interface (a form) would first validate that information before using it to create the account.

When a user interface is replaced with a SIF interface, the subscribing application might be left trusting the validation rules of the source system unless something is put between the two to make sure that data quality is being maintained.

For example, in an MIS, it might not be too important that learner’s birth date is reasonable – to the MIS, it is only important that a value is entered into the field and that it is a valid date. But, to a VLE it is important that it is valid as well as reasonable. If a user were entering this information by hand into the VLE, he or she would never be able to enter today’s date into the birth date field.

Veracity

Veracity is a product that does several things:

• It monitors data quality based on rules that have been set up at a certain level – for example, rules can be set up at the school level to capture errors as they come out of the MIS that will become problems in other systems that are connected to this system – checks like invalid birth dates, missing names, etc.
• It maintains a consolidated database of data collected from all systems that can be used for reporting or other data analysis
• It provides a portal interface so that users can see where they need to make changes to their data

Where Veracity Fits In

We recommend installing Veracity in two places in this architecture:

1. In the school zone – used here, the rule set that is applied to the collected data will check it against the needs of the other systems that will use its data. Additionally, the portal interface it provides will give users an insight into how close they are to the place where the data they enter can transparently flow throughout the system without alteration.
2. Post-Envoy – Envoy is the software that will match records and consolidate the information from the single zones into a group zone. Having Veracity subscribe here will provide a consolidated reporting and analysis database and a place where business rules can run that check for problems that might occur where data from one school collides with that from another.

The following diagram shows Veracity as it would be installed to monitor data quality for each of the school zones:

Adding Veracity – Click image to show in full screen

Since the ZIServer is being hosted, it is not unreasonable to assume that these copies of Veracity would be hosted as well.

*NOTE: In Group 6 items, only three objects are recommended for Derbyshire, but we are leaving room for four data quality checks. Adding them should add 8 additional hours (2 hours per check) to the configuration process and shouldn’t add any additional time to the time it takes to create an installation package. So, the data quality checks for Group 6 in total should add a total of 8 hours.

Merging/Matching Data

The primary function of the Envoy product in this environment is to resolve differences when a physical object is  represented in more than one MIS. This can happen when a learner attends more than one school, a teacher teaches at more than one school or a parent has children attending more than one school.

When this occurs, what envoy does is to create a “virtual object” by choosing the base object from the “more reliable” of the two and making it appear as if there was a single large MIS providing all the records from a single source. What this accomplishes for subscribing applications is to provide them with a simplified view of the learner population and centralizing the complex, matching logic where it can be independently tested and verified.

For more information on how this works, see: Managed Virtual Zones.

In our implementation, we will use the Envoy product to perform this function, match records and combine records for all the schools into a single zone as depicted by the following diagram:

Adding Envoy – Click image to show in full screen

Setting up Envoy is more difficult than the other products because, for each of the SIF objects that it handles there are rules that must be, at a minimum, reviewed by the customer to make sure that everyone is in agreement as to how the object is being handled. These rules fall into two categories:

1. How records are matched. Envoy can use a hierarchy of rules when matching – for example, when matching learner records, it can first try ULN, family name, given name, post code and birth date. Then if that doesn’t match it can try ULN, family name, given name, gender and birth date, then three more combinations before giving up and assuming that the two records are for different people.
2. How the “more reliable” base object is chosen when there is a choice. If a child attends two schools, there are two LearnerPersonal records – which one should a subscribing application receive as the “more reliable” of the two? Sometimes it is easy to choose (because the SIF specification has rules for choosing, other times is is more difficult).

With Envoy, most of the more difficult review work will be in the Group 1 objects. This is because Group 1 includes all of the “base” objects that are the ones that get matched: LAInfo, SchoolInfo, LearnerPersonal, ContactPersonal and WorkforcePersonal. These objects will have the hierarchy of matching rules that need to be reviewed where the other objects will have simpler, more definitive rules (however, they will typically have many more of them). This is why the bulk of the hours show up in the estimate for Group 1.

NOTE 1: In Group 6 items, only three objects are recommended for Derbyshire, but we are leaving room for four data quality checks. Adding them should add 8 additional hours (2 hours per check) to the configuration process and shouldn’t add any additional time to the time it takes to create an installation package. So, the data quality checks for Group 6 in total should add a total of 8 hours.

NOTE 2: At this point, it would be possible to connect a “Local Authority Level” SIF-enabled application to the consolidated zone (as we intend to do with Ipseity).

Identity and Role Management

Once records have been data quality checked and consolidated, information is passed to the Identity and Role Manager, Ipseity™ (formerly codename Sentry). This SIF agent will subscribe to several objects and publish one – by subscribing, it will receive all of the information it needs to create account names and most of the information it needs to assign roles to the applications it manages.

The following diagram gives an overall picture of where Ipseity fits into the overall architecture:

 Adding Ipseity – Click image to show in full screen

As shown in this picture, Ipseity has not only a SIF interface, but also a web services interface. This interface was put in so that non-SIF applications could easily use it features such as:

• Adding new users to the pool of users being managed, so that Ipseity can assign IDs that are consistent with the automatically managed IDs
• Query the database of all users for {new/changed/deleted} users, which users are assigned to which schools, which users are assigned to what roles (in which schools), etc..

There are several web service calls supported by Ipseity that all return recordsets, making it simple for other applications to access data in its store either from the same location or remotely (through HTTPS). For more information on these facilities, see: Extending Identity Management – Roles

Subscribed Objects

Ipseity will subscribe to the following objects:

• LAInfo
• SchoolInfo
• LearnerPersonal
• LearnerSchoolEnrolment
• WorkforcePersonal – this contains all information about teachers, including all schools at which they work. Some of these records may be skipped because their assignments might indicate that they have no computer interaction
• ContactPersonal – this contains basic information about a parent or other contact. Some of these records will be ignored, such as those for after school programs or medical facilities that are stored in an MIS, but will not be needed in Ipseity for Identity or Role management purposes
• LearnerContact – this links a contact record to a learner record

Published Object

Ipseity will publish the following object:

• Identity – this will contain the identities it assigns, including AD account names, Shibboleth names, and initial passwords, etc.

Roles Being Managed

At a minimum, roles for the products we are installing will need to be managed.

For example, ZIServer will need to be assigned people who will be able to create and manage zones, assign permissions and those who will have access to the audits. It might be the case that people in the schools may need access to the audits as well.

Veracity’s interface will need a similar administrator role, but it will also have interfaces for a person who creates business rules (someone who knows about school MIS data – probably not a system administrator), and several people in each school who will be using its interface to correct errors in data (these will probably not overlap with the other two at all).

Envoy will need an administrator, but will also need someone who oversees the business rules that govern how record matching is done and choices are made (this person is more of an analyst, not a systems administrator), but there will be nobody in the school who will have access.

Question: In the following spreadsheet, it was indicated that the following roles are required for use by one or more Dell applications:

Original Excel file with edits:Sims MIS role data fields with SIF code values.xls

Would it be desirable to use Ipseity to manage these roles along with the roles for ZIServer, Veracity and Envoy? Users would be able to use the same interface and the applications would have access to the common web services interface.

Note: the setup of this application does not depend on the SIF objects being configured, so it will not vary from implementation to implementation.

Note: Unless additional applications are being added between POC and Production, Ipseity should not need to be changed in this phase.

Consolidated Data Quality Checking / Reporting

The final step in the process (or this could be swapped with the IDRM step) will be to set up a copy of Veracity that consolidates all data from all schools (and Ipseity) into a single database and performs another set of data quality rules on them.

The data quality rules that are run on this set of data should not be checking for the simple things that should have been remedied at the school, but rather other types of errors that show up when data from one school is combined with that from another.

An example might be: for a given learner we might see attendance records showing him attending both his morning and afternoon schools the same morning of the same day and this is not supposed to happen. Or, we see two children with completely different demographic information in different schools, but they have the same ULN.

The following diagram shows where this copy of Veracity is installed in the architecture:

 Adding Veracity (on consolidated data set) – Click image to show in full screen

This copy becomes a subscriber in the combined zone. Since Veracity’s database is effectively an Enterprise Data Store, we’ve found that it is very useful to other applications such as:

• Data Warehouses: these can sample the database on given intervals. Typically, the DW builds a series of views, stored procedures or compiled queries that get called on a regular basis that take a snapshot from our repository.
• Reporting Application: tools such as SQL Reporting Services are used in districts to produced real-time reports on data that is collected from many sources. In our scenario, we are only collecting data from the MIS and from Ipseity, but in other places they may collect data from as many as 8 or 9 places.
• Data Analysis: some use SQL Analysis in similar ways to perform data analysis on the Veracity database or a replicated snapshot.

To view the database layout and to get an empty Veracity Database Backup, see: Veracity UK 1.2 Data

Summary

This is an estimate; these hours are based on certain assumptions about a number of business rules that will be requested and a level of data cleanliness we will find when we see the actual data.

This could vary widely – the difference in the effort required for us to successfully integrate all these applications between locations where the data is carefully maintained and those where it isn’t can be huge (up to 50% of the project).  This is why we suggest a step-by-step approach where the data quality issues are addressed in the first step, so that they will not haunt us in later steps.

We also use the strategy of giving tools to users in the first step that “make their life better”. One would think that Veracity “looking over their shoulder” would not fit into that category, but it can.

For example, if we were to encode their annual census rules into quality checks, it would give them a tool that allows them to see how close they are to not having any work to do next time a census is due – except to run a query against their local Veracity DB and export it – perhaps…

These estimates assume that the work would be able to be performed from our office through a connection such as a VPN tunnel. We found that working offsite (such as we did during the first POC) added about 80% to the length of time it took to do a given task (so 10 hours in our office working through a remote facility = 18 hours working at the Dell site).

This SOW is intended to be a working document – the comments facility is here to allow other team members to pose questions, make comments and give suggestions – please use it. Comments will first go through me (Rob Hutchison) for approval, so if you don’t want it to be seen by everyone, put “PRIVATE” on the first line and I won’t publish it for all. Thanks.

Introduction

In June 2009, Dell, Visual Software and ITWorx assembled a Proof of Concept for the Cambridgeshire BSF bid and demonstrated that the Schools Interoperability Framework could be used to transport learner information such as that which would come from a typical UK MIS to a learning platform in near real time to meet the requirements of the Local Authority (LA).

In the first phase, for several reasons, we used a MIS simulator instead of using a copy of Capita SIMS, Serco CMIS or one of the other MIS systems.  In the second phase of this overall project, we will be putting together a lasting platform that will be used in the Cambridgeshire and Derbyshire implementations.

This document’s first purpose is to supply Visual Software’s recommendations for setting up a SIF implementation for these schools, connecting the MIS instances, providing matching, identity and role management, providing facilities for checking the data for quality standards and providing an interface for the other Dell-incorporated software. In addition, its second purpose is to provide cost estimates for the work that will be involved in this integration project.

Visual Software Component Overview

Visual Software components will interface with instances of Management Information Systems at schools and with Dell supplied components within the data center. In general, our components will be connected as follows:

Overview – Click image to show in full screen

Staged Implementation

It has been indicated that, for the two implementations there may or may not be a Proof of Concept stage and that the two implementations themselves not include the exact same SIF coverage.  To make what we offer as flexible as possible, we shall split the UK SIF specification into a set of object groups. For each of the combinations of (implementation site and {POC and Production}), we will make recommendations as which groups would reasonably fall under the combination.

So, in the case of this document we will highlight our recommendation for the Derbyshire Proof of Concept, while leaving grayed out things that could be added as optional elements. We will document other combinations separately.

SIF Object Groups

The UK SIF specification does not group objects into clusters of any sort, but for the purposes of these documents, we will group them as follows:

• Basic: those objects needed for identity & role management, including learners, teachers and parents. It includes the schools that people with which people are associated, but not the school groups.
• Learner Groups: this extends the basic set to extend learner and teacher information to include learner group information. It might be argued that this group should be done at the same time as Timetabling, but a great deal of useful information can be derived from only these objects without the timetabling objects.
• Assessments: a group of interdependent objects that store information about assessments
• Attendance: these relay attendance information, either in summary or detailed information – in the UK, it is typical to send complete information (not just absences), so expect to receive a high volume of information
• Timetabling: this is information relative to how the school groups were established
• Uncategorized: these objects don’t fall into any of the above categories, but do include some important messages such as behavior, meal status and special education objects.

The following table shows which SIF objects are in each of these categories:

Recommendation for Derbyshire POC

Since the goal of this POC is to implement Identity and Role management and to supply Forefront Identity Manager with the basic information it needs to create and manage accounts, we recommend that we implement “Group 1: Basic” SIF objects in this phase.

Connecting MIS Systems

The following diagram gives an overview of how the MIS systems will be connected into the SIF infrastructure:

On the Zone Integration Server, a separate zone will be created for each school (for more information on the use of Zones, see SIF Zones) and each school’s copy of Mimic will be set as the Provider of learner information in that zone. Mimic will generate SIF messages by looking at the CSV files that each copy of SIMS creates on a scheduled basis, comparing it against the copy it looked at on its last scheduled run. (For more information on how Mimic works, see Mimic Users Guide)

NOTE: A single ZIServer can host many (even thousands of) zones, and these zones can either be local (HTTP) or remote (HTTPS).

In order for the Mimic software to run, there will need to be a Windows computer running it and storing its SQL Server database in each location where there is an MIS.

Mimic as a Web Site

Like all SIF Agents, Mimic runs as a web site. Like many SIF Agents, Mimic may run in SIF Pull or Push mode (Pull mode is analogous to batch mode and Push mode is similar to real-time mode – the meanings are very different than the typical meanings of ‘Push’ and ‘Pull’ in English. For a more complete description, see SIF Push and Pull Modes.)

This has implications for the installation because SIF Push mode (real-time) will require that the installation at the school have an external IP number and that the ZIS be able to communicate with the Agent. Although this is more work, we recommend this configuration because it allows the school to have its own certificate and allows us to set up Encryption Level 4, Authentication Level 3 where each side verifies the certificate of the other.

It is not necessary to use an outside certificate authority (you could use Certificate Services) because we will be setting up a closed community of servers. NOTE: In E2BN, we found out that, although it is supposed to be a simple matter to get certificates whenever we needed them, in reality it wasn’t all that simple.

Mimic Database

We typically use SQL Server Express Edition for Mimic installations. We are recommending installing SQL Server 2008 R2, especially since it now expands to 10GB and allows users to back up and restore.

In this estimate, the length of time it would take to create and test the installation package will not change significantly with the differing number of SIF objects implemented; most of the time in doing that will be in the setup, testing and making sure that the correct results were achieved when the tests were run. The configuration of the Mimic product, however, will largely depend on the number of objects being configured.

Managing School Data Quality

Normally, applications have user interfaces that prevent things from automatically happening when it receives “bad” information. For example, an application that creates user accounts from information entered into a user interface (a form) would first validate that information before using it to create the account.

When a user interface is replaced with a SIF interface, the subscribing application might be left trusting the validation rules of the source system unless something is put between the two to make sure that data quality is being maintained.

For example, in an MIS, it might not be too important that learner’s birth date is reasonable – to the MIS, it is only important that a value is entered into the field and that it is a valid date. But, to a VLE it is important that it is valid as well as reasonable. If a user were entering this information by hand into the VLE, he or she would never be able to enter today’s date into the birth date field.

Veracity

Veracity is a product that does several things:

• It monitors data quality based on rules that have been set up at a certain level – for example, rules can be set up at the school level to capture errors as they come out of the MIS that will become problems in other systems that are connected to this system – checks like invalid birth dates, missing names, etc.
• It maintains a consolidated database of data collected from all systems that can be used for reporting or other data analysis
• It provides a portal interface so that users can see where they need to make changes to their data

Where Veracity Fits In

We recommend installing Veracity in two places in this architecture:

1. In the school zone – used here, the rule set that is applied to the collected data will check it against the needs of the other systems that will use its data. Additionally, the portal interface it provides will give users an insight into how close they are to the place where the data they enter can transparently flow throughout the system without alteration.
2. Post-Envoy – Envoy is the software that will match records and consolidate the information from the single zones into a group zone. Having Veracity subscribe here will provide a consolidated reporting and analysis database and a place where business rules can run that check for problems that might occur where data from one school collides with that from another.

The following diagram shows Veracity as it would be installed to monitor data quality for each of the school zones:

Adding Veracity – Click image to show in full screen

Since the ZIServer is being hosted, it is not unreasonable to assume that these copies of Veracity would be hosted as well.

Merging/Matching Data

The primary function of the Envoy product in this environment is to resolve differences when a physical object is  represented in more than one MIS. This can happen when a learner attends more than one school, a teacher teaches at more than one school or a parent has children attending more than one school.

When this occurs, what envoy does is to create a “virtual object” by choosing the base object from the “more reliable” of the two and making it appear as if there was a single large MIS providing all the records from a single source. What this accomplishes for subscribing applications is to provide them with a simplified view of the learner population and centralizing the complex, matching logic where it can be independently tested and verified.

For more information on how this works, see: Managed Virtual Zones.

In our implementation, we will use the Envoy product to perform this function, match records and combine records for all the schools into a single zone as depicted by the following diagram:

Adding Envoy – Click image to show in full screen

Setting up Envoy is more difficult than the other products because, for each of the SIF objects that it handles there are rules that must be, at a minimum, reviewed by the customer to make sure that everyone is in agreement as to how the object is being handled. These rules fall into two categories:

1. How records are matched. Envoy can use a hierarchy of rules when matching – for example, when matching learner records, it can first try ULN, family name, given name, post code and birth date. Then if that doesn’t match it can try ULN, family name, given name, gender and birth date, then three more combinations before giving up and assuming that the two records are for different people.
2. How the “more reliable” base object is chosen when there is a choice. If a child attends two schools, there are two LearnerPersonal records – which one should a subscribing application receive as the “more reliable” of the two? Sometimes it is easy to choose (because the SIF specification has rules for choosing, other times is is more difficult).

With Envoy, most of the more difficult review work will be in the Group 1 objects. This is because Group 1 includes all of the “base” objects that are the ones that get matched: LAInfo, SchoolInfo, LearnerPersonal, ContactPersonal and WorkforcePersonal. These objects will have the hierarchy of matching rules that need to be reviewed where the other objects will have simpler, more definitive rules (however, they will typically have many more of them). This is why the bulk of the hours show up in the estimate for Group 1.

NOTE: At this point, it would be possible to connect a “Local Authority Level” SIF-enabled application to the consolidated zone (as we intend to do with Ipseity).

Identity and Role Management

Once records have been data quality checked and consolidated, information is passed to the Identity and Role Manager, Ipseity™ (formerly codename Sentry). This SIF agent will subscribe to several objects and publish one – by subscribing, it will receive all of the information it needs to create account names and most of the information it needs to assign roles to the applications it manages.

The following diagram gives an overall picture of where Ipseity fits into the overall architecture:

 Adding Ipseity – Click image to show in full screen

As shown in this picture, Ipseity has not only a SIF interface, but also a web services interface. This interface was put in so that non-SIF applications could easily use it features such as:

• Adding new users to the pool of users being managed, so that Ipseity can assign IDs that are consistent with the automatically managed IDs
• Query the database of all users for {new/changed/deleted} users, which users are assigned to which schools, which users are assigned to what roles (in which schools), etc..

There are several web service calls supported by Ipseity that all return recordsets, making it simple for other applications to access data in its store either from the same location or remotely (through HTTPS). For more information on these facilities, see: Extending Identity Management – Roles

Subscribed Objects

Ipseity will subscribe to the following objects:

• LAInfo
• SchoolInfo
• LearnerPersonal
• LearnerSchoolEnrolment
• WorkforcePersonal – this contains all information about teachers, including all schools at which they work. Some of these records may be skipped because their assignments might indicate that they have no computer interaction
• ContactPersonal – this contains basic information about a parent or other contact. Some of these records will be ignored, such as those for after school programs or medical facilities that are stored in an MIS, but will not be needed in Ipseity for Identity or Role management purposes
• LearnerContact – this links a contact record to a learner record

Published Object

Ipseity will publish the following object:

• Identity – this will contain the identities it assigns, including AD account names, Shibboleth names, and initial passwords, etc.

Roles Being Managed

At a minimum, roles for the products we are installing will need to be managed.

For example, ZIServer will need to be assigned people who will be able to create and manage zones, assign permissions and those who will have access to the audits. It might be the case that people in the schools may need access to the audits as well.

Veracity’s interface will need a similar administrator role, but it will also have interfaces for a person who creates business rules (someone who knows about school MIS data – probably not a system administrator), and several people in each school who will be using its interface to correct errors in data (these will probably not overlap with the other two at all).

Envoy will need an administrator, but will also need someone who oversees the business rules that govern how record matching is done and choices are made (this person is more of an analyst, not a systems administrator), but there will be nobody in the school who will have access.

Question: In the following spreadsheet, it was indicated that the following roles are required for use by one or more Dell applications:

Original Excel file with edits:Sims MIS role data fields with SIF code values.xls

Would it be desirable to use Ipseity to manage these roles along with the roles for ZIServer, Veracity and Envoy? Users would be able to use the same interface and the applications would have access to the common web services interface.

Note: the setup of this application does not depend on the SIF objects being configured, so it will not vary from implementation to implementation.

Consolidated Data Quality Checking / Reporting

The final step in the process (or this could be swapped with the IDRM step) will be to set up a copy of Veracity that consolidates all data from all schools (and Ipseity) into a single database and performs another set of data quality rules on them.

The data quality rules that are run on this set of data should not be checking for the simple things that should have been remedied at the school, but rather other types of errors that show up when data from one school is combined with that from another.

An example might be: for a given learner we might see attendance records showing him attending both his morning and afternoon schools the same morning of the same day and this is not supposed to happen. Or, we see two children with completely different demographic information in different schools, but they have the same ULN.

The following diagram shows where this copy of Veracity is installed in the architecture:

 Adding Veracity (on consolidated data set) – Click image to show in full screen

This copy becomes a subscriber in the combined zone. Since Veracity’s database is effectively an Enterprise Data Store, we’ve found that it is very useful to other applications such as:

• Data Warehouses: these can sample the database on given intervals. Typically, the DW builds a series of views, stored procedures or compiled queries that get called on a regular basis that take a snapshot from our repository.
• Reporting Application: tools such as SQL Reporting Services are used in districts to produced real-time reports on data that is collected from many sources. In our scenario, we are only collecting data from the MIS and from Ipseity, but in other places they may collect data from as many as 8 or 9 places.
• Data Analysis: some use SQL Analysis in similar ways to perform data analysis on the Veracity database or a replicated snapshot.

To view the database layout and to get an empty Veracity Database Backup, see: Veracity UK 1.2 Data

Although there is no data warehouse requirement for Derbyshire, data quality checking is an essential part of this implementation. The above estimates are for that data quality checking and not to support a data warehouse interface.

Summary

This is an estimate; these hours are based on certain assumptions about a number of business rules that will be requested and a level of data cleanliness we will find when we see the actual data.

This could vary widely – the difference in the effort required for us to successfully integrate all these applications between locations where the data is carefully maintained and those where it isn’t can be huge (up to 50% of the project).  This is why we suggest a step-by-step approach where the data quality issues are addressed in the first step, so that they will not haunt us in later steps.

We also use the strategy of giving tools to users in the first step that “make their life better”. One would think that Veracity “looking over their shoulder” would not fit into that category, but it can.

For example, if we were to encode their annual census rules into quality checks, it would give them a tool that allows them to see how close they are to not having any work to do next time a census is due – except to run a query against their local Veracity DB and export it – perhaps…

These estimates assume that the work would be able to be performed from our office through a connection such as a VPN tunnel. We found that working offsite (such as we did during the first POC) added about 80% to the length of time it took to do a given task (so 10 hours in our office working through a remote facility = 18 hours working at the Dell site).

This SOW is intended to be a working document – the comments facility is here to allow other team members to pose questions, make comments and give suggestions – please use it. Comments will first go through me (Rob Hutchison) for approval, so if you don’t want it to be seen by everyone, put “PRIVATE” on the first line and I won’t publish it for all. Thanks.

A “zone” in SIF is a logical place or container where two or more agents can share data, isolated from data in any other zone. There can be many zones hosted on a typical Zone Integration Server and one agent may be registered in more than one zone concurrently (but to be useful at all it must be registered in at least one zone).

Zone Architecture

To understand where zones will be needed in an architecture, the needs and capabilities of the subscribing applications that will be used must be properly considered.

The reason why this is necessary is that, on occasion (when initializing, at end-of-year rollover and if they the want to re-sync) , subscribers will make a “request all” for certain object types. What they will get in return are all records of that type in the zone.

• Is this application allowed to see all the information returned? For example, if this application was a library system installed at a school and the information returned to it contained information from other schools, then the answer would be “no”. One school should only receive information for students from that school.

It has been a common practice to create a zone for each Student Management System (or SIS (US), or MIS (UK)); sometimes this works well , other times it doesn’t. Ideally, it should probably be more like: create a zone at the lowest level where there will be a subscribing SIF application.

Zone Architecture Examples

The following sections give some examples of SIF zone architectures that might be appropriate, depending on where student information is managed and where it is needed by subscribing applications.

United Kingdom Example

In the UK, data is typically managed in Management Information Systems (MIS) that are installed and run from individual schools. The Zone Integration Servers may reside at the school or may be centrally hosted and the subscribing applications may run locally from the schools, from a local authority or may be centrally hosted from a Regional Broadband Consortium (RBC).

In the UK, when a learner reaches age 14, he or she may begin to attend more than one school during the course of a day, causing his or her records to be managed in more than one of these MIS systems. It shouldn’t be assumed that the SIF RefIds from any of these MIS systems are synchronized, so it should also be assumed that if a learner attends two different schools, his or her school records will look like those from two different people.

NOTE: technically, the same thing will happen for special education learners who attend more than one school – their records will be maintained in more than one MIS.

(click image to open in separate window)

In this example, three schools each have their own independent zone (they could be hosted on the same Zone Integration Server) containing information provided by the school’s MIS. The subscribing application, also installed at the school, gets its information from the zone through its SIF agent. Because of the way the ZIS manages zones, it is guaranteed that information from one zone will never be mixed with that from another zone.

UK Example – Shared Applications

The above example is simple, but not all applications are installed at schools – many are hosted either at a Local Authority or at an RBC and shared by several schools.

This presents a problem. One of the most basic rules in SIF is that in a given zone, there may be only a single provider for each object.  This makes sense since a provider is defined as the agent who, by default, receives requests for a given object when a subscriber agent makes a request and doesn’t specify where to send the request (it is sent to the provider agent).

So, what you can’t do is to have all of the MIS (SIS) agents be providers in a common zone. But, there are alternatives:

• Multi-zoned agents: these are agents that register in more than one zone. For subscribing applications, writing one of these agents is somewhat difficult; for publishers it is very difficult. These agents must be able to keep track of many sets of RefId values, one for each school and if a learner appears in more than one school, it gets even more complex.
• Managed Virtual Zones (MVZ): when Visual Software entered the UK market, it realized that a problem existed with currently available facilities using SIF as implemented with multi-zoned agents. To address these problems, it invented Managed Virtual Zones and created the Envoy product to implement it. With this architecture, the application agents do not need to be written with the complex multi-zone logic – all of the complexities are centralized in one place and can be tested independently and verified before other agents are connected.

The following is a picture of how applications would be connected using a MVZ architecture and Envoy:

(click image to open in separate window)

In this example, the subscriber in the combined zone would not need to know any details about who is providing what object or learners that go to two different schools – the details would be “ironed out” and the combined community would appear as if it was being maintained by a single MIS (SIS). For more information on Envoy and MVZ, see Managed Virtual Zones.

Introduction

In June 2009, Dell, Visual Software and ITWorx assembled a Proof of Concept for the Cambridgeshire BSF bid and demonstrated that the Schools Interoperability Framework could be used to transport learner information such as that which would come from a typical UK MIS to a learning platform in near real time to meet the requirements of the Local Authority (LA).

In the first phase, for several reasons, we used a MIS simulator instead of using a copy of Capita SIMS, Serco CMIS or one of the other MIS systems.  In the second phase of this overall project, we will be putting together a lasting platform that will be used in the Cambridgeshire and Derbyshire implementations.

This document’s first purpose is to supply Visual Software’s recommendations for setting up a SIF implementation for these schools, connecting the MIS instances, providing matching, identity and role management, providing facilities for checking the data for quality standards and providing an interface for the other Dell-incorporated software. In addition, its second purpose is to provide cost estimates for the work that will be involved in this integration project.

Visual Software Component Overview

Visual Software components will interface with instances of Management Information Systems at schools and with Dell supplied components within the data center. In general, our components will be connected as follows:

Overview – Click image to show in full screen

Connecting MIS Systems

The following diagram gives an overview of how the MIS systems will be connected into the SIF infrastructure:

On the Zone Integration Server, a separate zone will be created for each school (for more information on the use of Zones, see SIF Zones) and each school’s copy of Mimic will be set as the Provider of learner information in that zone. Mimic will generate SIF messages by looking at the CSV files that each copy of SIMS creates on a scheduled basis, comparing it against the copy it looked at on its last scheduled run. (For more information on how Mimic works, see Mimic Users Guide)

NOTE: A single ZIServer can host many (even thousands of) zones, and these zones can either be local (HTTP) or remote (HTTPS).

In order for the Mimic software to run, there will need to be a Windows computer running it and storing its SQL Server database in each location where there is an MIS.

Mimic as a Web Site

Like all SIF Agents, Mimic runs as a web site. Like many SIF Agents, Mimic may run in SIF Pull or Push mode (Pull mode is analogous to batch mode and Push mode is similar to real-time mode – the meanings are very different than the typical meanings of ‘Push’ and ‘Pull’ in English. For a more complete description, see SIF Push and Pull Modes.)

This has implications for the installation because SIF Push mode (real-time) will require that the installation at the school have an external IP number and that the ZIS be able to communicate with the Agent. Although this is more work, we recommend this configuration because it allows the school to have its own certificate and allows us to set up Encryption Level 4, Authentication Level 3 where each side verifies the certificate of the other.

It is not necessary to use an outside certificate authority (you could use Certificate Services) because we will be setting up a closed community of servers. NOTE: In E2BN, we found out that, although it is supposed to be a simple matter to get certificates whenever we needed them, in reality it wasn’t all that simple.

Mimic Database

We typically use SQL Server Express Edition for Mimic installations. We are recommending installing SQL Server 2008 R2, especially since it now expands to 10GB and allows users to back up and restore.

Managing School Data Quality

Normally, applications have user interfaces that prevent things from automatically happening when it receives “bad” information. For example, an application that creates user accounts from information entered into a user interface (a form) would first validate that information before using it to create the account.

When a user interface is replaced with a SIF interface, the subscribing application might be left trusting the validation rules of the source system unless something is put between the two to make sure that data quality is being maintained.

For example, in an MIS, it might not be too important that learner’s birth date is reasonable – to the MIS, it is only important that a value is entered into the field and that it is a valid date. But, to a VLE it is important that it is valid as well as reasonable. If a user were entering this information by hand into the VLE, he or she would never be able to enter today’s date into the birth date field.

Veracity

Veracity is a product that does several things:

• It monitors data quality based on rules that have been set up at a certain level – for example, rules can be set up at the school level to capture errors as they come out of the MIS that will become problems in other systems that are connected to this system – checks like invalid birth dates, missing names, etc.
• It maintains a consolidated database of data collected from all systems that can be used for reporting or other data analysis
• It provides a portal interface so that users can see where they need to make changes to their data

Where Veracity Fits In

We recommend installing Veracity in two places in this architecture:

1. In the school zone – used here, the rule set that is applied to the collected data will check it against the needs of the other systems that will use its data. Additionally, the portal interface it provides will give users an insight into how close they are to the place where the data they enter can transparently flow throughout the system without alteration.
2. Post-Envoy – Envoy is the software that will match records and consolidate the information from the single zones into a group zone. Having Veracity subscribe here will provide a consolidated reporting and analysis database and a place where business rules can run that check for problems that might occur where data from one school collides with that from another.

The following diagram shows Veracity as it would be installed to monitor data quality for each of the school zones:

Adding Veracity – Click image to show in full screen

Since the ZIServer is being hosted, it is not unreasonable to assume that these copies of Veracity would be hosted as well.

Merging/Matching Data

The primary function of the Envoy product in this environment is to resolve differences when a physical object is  represented in more than one MIS. This can happen when a learner attends more than one school, a teacher teaches at more than one school or a parent has children attending more than one school.

When this occurs, what envoy does is to create a “virtual object” by choosing the base object from the “more reliable” of the two and making it appear as if there was a single large MIS providing all the records from a single source. What this accomplishes for subscribing applications is to provide them with a simplified view of the learner population and centralizing the complex, matching logic where it can be independently tested and verified.

For more information on how this works, see: Managed Virtual Zones.

In our implementation, we will use the Envoy product to perform this function, match records and combine records for all the schools into a single zone as depicted by the following diagram:

Adding Envoy – Click image to show in full screen

Setting up Envoy is more difficult than the other products because, for each of the SIF objects that it handles there are rules that must be, at a minimum, reviewed by the customer to make sure that everyone is in agreement as to how the object is being handled. These rules fall into two categories:

1. How records are matched. Envoy can use a hierarchy of rules when matching – for example, when matching learner records, it can first try ULN, family name, given name, post code and birth date. Then if that doesn’t match it can try ULN, family name, given name, gender and birth date, then three more combinations before giving up and assuming that the two records are for different people.
2. How the “more reliable” base object is chosen when there is a choice. If a child attends two schools, there are two LearnerPersonal records – which one should a subscribing application receive as the “more reliable” of the two? Sometimes it is easy to choose (because the SIF specification has rules for choosing, other times is is more difficult).

NOTE: At this point, it would be possible to connect a “Local Authority Level” SIF-enabled application to the consolidated zone (as we intend to do with Ipseity).

Identity and Role Management

Once records have been data quality checked and consolidated, information is passed to the Identity and Role Manager, Ipseity™ (formerly codename Sentry). This SIF agent will subscribe to several objects and publish one – by subscribing, it will receive all of the information it needs to create account names and most of the information it needs to assign roles to the applications it manages.

The following diagram gives an overall picture of where Ipseity fits into the overall architecture:

 Adding Ipseity – Click image to show in full screen

As shown in this picture, Ipseity has not only a SIF interface, but also a web services interface. This interface was put in so that non-SIF applications could easily use it features such as:

• Adding new users to the pool of users being managed, so that Ipseity can assign IDs that are consistent with the automatically managed IDs
• Query the database of all users for {new/changed/deleted} users, which users are assigned to which schools, which users are assigned to what roles (in which schools), etc..

There are several web service calls supported by Ipseity that all return recordsets, making it simple for other applications to access data in its store either from the same location or remotely (through HTTPS). For more information on these facilities, see: Extending Identity Management – Roles

Subscribed Objects

Ipseity will subscribe to the following objects:

• LAInfo
• SchoolInfo
• LearnerPersonal
• LearnerSchoolEnrolment
• WorkforcePersonal – this contains all information about teachers, including all schools at which they work. Some of these records may be skipped because their assignments might indicate that they have no computer interaction
• ContactPersonal – this contains basic information about a parent or other contact. Some of these records will be ignored, such as those for after school programs or medical facilities that are stored in an MIS, but will not be needed in Ipseity for Identity or Role management purposes
• LearnerContact – this links a contact record to a learner record

Published Object

Ipseity will publish the following object:

• Identity – this will contain the identities it assigns, including AD account names, Shibboleth names, and initial passwords, etc.

Roles Being Managed

At a minimum, roles for the products we are installing will need to be managed.

For example, ZIServer will need to be assigned people who will be able to create and manage zones, assign permissions and those who will have access to the audits. It might be the case that people in the schools may need access to the audits as well.

Veracity’s interface will need a similar administrator role, but it will also have interfaces for a person who creates business rules (someone who knows about school MIS data – probably not a system administrator), and several people in each school who will be using its interface to correct errors in data (these will probably not overlap with the other two at all).

Envoy will need an administrator, but will also need someone who oversees the business rules that govern how record matching is done and choices are made (this person is more of an analyst, not a systems administrator), but there will be nobody in the school who will have access.

Question: In the following spreadsheet, it was indicated that the following roles are required for use by one or more Dell applications:

Original Excel file with edits:Sims MIS role data fields with SIF code values.xls

Would it be desirable to use Ipseity to manage these roles along with the roles for ZIServer, Veracity and Envoy? Users would be able to use the same interface and the applications would have access to the common web services interface.

Consolidated Data Quality Checking / Reporting

The final step in the process (or this could be swapped with the IDRM step) will be to set up a copy of Veracity that consolidates all data from all schools (and Ipseity) into a single database and performs another set of data quality rules on them.

The data quality rules that are run on this set of data should not be checking for the simple things that should have been remedied at the school, but rather other types of errors that show up when data from one school is combined with that from another.

An example might be: for a given learner we might see attendance records showing him attending both his morning and afternoon schools the same morning of the same day and this is not supposed to happen. Or, we see two children with completely different demographic information in different schools, but they have the same ULN.

The following diagram shows where this copy of Veracity is installed in the architecture:

 Adding Veracity (on consolidated data set) – Click image to show in full screen

This copy becomes a subscriber in the combined zone. Since Veracity’s database is effectively an Enterprise Data Store, we’ve found that it is very useful to other applications such as:

• Data Warehouses: these can sample the database on given intervals. Typically, the DW builds a series of views, stored procedures or compiled queries that get called on a regular basis that take a snapshot from our repository.
• Reporting Application: tools such as SQL Reporting Services are used in districts to produced real-time reports on data that is collected from many sources. In our scenario, we are only collecting data from the MIS and from Ipseity, but in other places they may collect data from as many as 8 or 9 places.
• Data Analysis: some use SQL Analysis in similar ways to perform data analysis on the Veracity database or a replicated snapshot.

To view the database layout and to get an empty Veracity Database Backup, see: Veracity UK 1.2 Data

Summary

This is an estimate; these hours are based on certain assumptions about a number of business rules that will be requested and a level of data cleanliness we will find when we see the actual data.

This could vary widely – the difference in the effort required for us to successfully integrate all these applications between locations where the data is carefully maintained and those where it isn’t can be huge (up to 50% of the project).  This is why we suggest a step-by-step approach where the data quality issues are addressed in the first step, so that they will not haunt us in later steps.

We also use the strategy of giving tools to users in the first step that “make their life better”. One would think that Veracity “looking over their shoulder” would not fit into that category, but it can.

For example, if we were to encode their annual census rules into quality checks, it would give them a tool that allows them to see how close they are to not having any work to do next time a census is due – except to run a query against their local Veracity DB and export it – perhaps…

These estimates assume that the work would be able to be performed from our office through a connection such as a VPN tunnel. We found that working offsite (such as we did during the first POC) added about 80% to the length of time it took to do a given task (so 10 hours in our office working through a remote facility = 18 hours working at the Dell site).

This SOW is intended to be a working document – the comments facility is here to allow other team members to pose questions, make comments and give suggestions – please use it. Comments will first go through me (Rob Hutchison) for approval, so if you don’t want it to be seen by everyone, put “PRIVATE” on the first line and I won’t publish it for all. Thanks.