311 Phase 2 Run Book

This document outlines the required documented need for project transition into the service delivery team.  Project to Operation transition begins with a document analysis. Information should be forwarded to allow the service team to review and ask questions around clarification. In cases, where documentation is not available, the project team will work with the service team via knowledge sharing workshops to address any gaps.

 

DOCUMENTATIONS/DELIVERABLES: 

Documentation type

Available

Y/N

Document format

Document location

Documentation type

Available

Y/N

Document format

Document location

Project Charter

 

 

311 P1 Project charter to be used, new governance structure/body

 

 

High Level Business Requirements

 

 JIRA

User Stories , Confluence page summary 

Required Confluence page with a summary of all features in the MVP delivered in final release.

Each Epic/feature can drill down to user stories, tasks etc. which were approved and delivered  

 

  https://toronto.atlassian.net/secure/Dashboard.jspa?selectPageId=11258

 

 

Detailed Business Requirements

 

 

 

o   High Level Bus.Reqmts: https://toronto.atlassian.net/wiki/spaces/EP2/pages/421134364

§  Ref Appendix J.2

§  Appendix V (Spark recommendation included in the RFP)

o   User Stories                 : https://toronto.atlassian.net/secure/Dashboard.jspa?selectPageId=11258

o   Project Docs                : ECRM - Phase II- 311 Toronto CRM Project - Confluence (atlassian.net)

 

Managing Appointment Blocks

Service Request Categories (Updated document that includes comments from CCG and 311 responses (https://toronto.atlassian.net/browse/EP2-5546 )

 

Special Characters: Special Characters

RegExr: Learn, Build, & Test RegEx

 

Business Impact Assessment

 

 

Documentation of impact assessment if any conducted for any major change. Feature, product or deliverable

 

JIRA Tasks labelled as Decisions  ( Shine/Roopy)

 

The link to the EARB minutes can be found at https://toronto.atlassian.net/browse/EP2-1687 .

Attached Email for below decision.

  1. Oracle Database was the selected temporary option with migration plan to Postgres for Lagan data migration. EARB suggested to consider the user experience of keeping data in Oracle and move to enterprise-wide Archiving Solution when it is in place.

 

Documentation of Cloud assessment and approval

 

 

Email or Documentation with assessment details and final approval from Cloud & Internet Services Team

 

Assessment from 311 P1 used ( Documentation from 2018)

CARB review not done as CARB was created only in 2019 ( Shine/Roopy to send emails related to this decision)

 

Architectural Design Documents

 

Confluence

Is Confluence page to be referred for Architectural details?

 

Require updated information to reflect new system architecture delivered by project.

Require documentation, diagram depicting the architectural components, integration with external systems etc.

 

Document 'Toronto Architecture': the architectural components /information seem to be related to current state (311p1).

https://toronto.atlassian.net/wiki/spaces/EP2/pages/1865678863

 

Below link has Webex recordings only.

https://toronto.atlassian.net/wiki/spaces/EP2/pages/470319198  

Functional Design Documents

 

 

 

Functional Specifications or Wireframes/Mock up's provided by 311 PMO team

Link:

 

·        Salesforce SPARK                    : https://toronto.atlassian.net/l/c/XD0FyNu8

 

 

Data Model

 

 

Logical and Physical Data Model documentation required.

·        Data Model & ERD                 : https://toronto.atlassian.net/l/c/3y03a1bH

 

Data Model – Design Document   – is this the most recent version?

https://toronto.atlassian.net/wiki/spaces/EP2/pages/1865678863

Technical Design Documents

 

 

 

Design Elements:  Technical Design Documentation should include details for all custom Salesforce components. Example: API Name, purpose, working details etc. to be included against each component.

 

Sustainment Team - Session #4 - Custom Code

Custom Code

Apex

Triggers

Flow-Invocable Classes

Batchable/Schedulable Classes

Flows

Utilized Services (GCC services etc.)

Lightning Web Components

Process Builders

Email templates

Validation Rules

Custom Metadata

Custom Settings

Mobile Application custom development

Global Configurations etc.

Scheduled jobs, batches etc.

 

·        Adding a SmartLink to an Article

·        Thesaurus Solution

·        Omni-Channel

·        Reports and Dashboards Questions from IBD

·        Regex pattern: Sustainment Team - Session #2 - Solution & Self-Service Configuration - ECRM - Phase II- 311 Toronto CRM Project - Confluence (atlassian.net)

·        P1 to P2 Field Mapping (MLS Noise Field Mapping):  https://toronto.atlassian.net/wiki/spaces/EP2/pages/1726775385

·        https://toronto.atlassian.net/wiki/spaces/EP2/pages/1726775385/MLS+Noise+Field+Mapping?preview=%2F1726775385%2F1963065465%2FSalesforce%20-%20MLS%20P1%20Field%20Mapping%20V1.4.xlsx

 

Field Validation: https://toronto.atlassian.net/wiki/spaces/EP2/pages/1463877732

 

 

Integrations

 

 

Middleware API endpoints for web, mobile and Salesforce

Integrations

 

Testing: https://app.swaggerhub.com/apis-docs/cityoftoronto/ECRM_311_to_backend/0.20.1#/Downstream%20Requests/downstreamUpdateRequest

 

https://toronto.atlassian.net/wiki/spaces/EP2/pages/2173861909

 

https://toronto.atlassian.net/wiki/spaces/EP2/pages/2001404015

 

https://toronto.atlassian.net/wiki/spaces/EP2/pages/1865678863

 

Certificate and Mutual Authentication

Certs installed  2

Mutual authentication  2 certs

Integration cert installed on downstream systems

Connected apps 311 Integration

 

Certificates – Downstream ( for connection with Divisional systems)

Mutual Auth – Upstream ( WP)

 

The permanent links to the API specifications are:

 https://app.swaggerhub.com/apis-docs/cityoftoronto/ECRM_311_to_backend/

https://app.swaggerhub.com/apis-docs/cityoftoronto/ECRM_Backend_to_311/

 

Service callout to Geocoder ( PIS/GCC) –  

Middleware API endpoints for web, mobile and Salesforce

Phase 1 - Data Migration Requirements and Mapping

QA Integration Testing

Process for Data Load involving Requests - ECRM - Phase II- 311 Toronto CRM Project - Confluence (atlassian.net)

 

Documentation of change to architectural or technical design

 

 

 

 

Documentation of Prototype

 

 

 

Documentation of Prototype feedback/ approval

 

 

 

Data Dictionary (includes all relevant objects, fields and data types)

 

 

Field and data type information from Salesforce to be extracted and provided in Excel format.

 

 

Test Plan(s)

 

 

Salesforce Unit Test plan, related documentation for each user story or deliverable.  

 

Xray Link:

Code to be transitioned to City Gitlab ?

Unit Test Scripts

 

 

Salesforce Apex Test scripts, Overall coverage, test classes, etc.

 

Unit Test Results

 

 

Salesforce Unit test results documentation  - custom code

System Integration Test Scripts

 

 

Integration Test Scripts  documentation  

 

System Integration Test Results

 

 

Integration Test results  documentation  

 

Performance Test Scripts

 

 

Performance Test Scripts  documentation  

 

Performance Test Results

 

 

Performance Test Results  documentation  

 

Action Item: follow up with Marco and Johan

 

Capacity

 

 

Data storage -  growth plan ( volume forecasted)

·        Additional Storage Analysis    : https://toronto.atlassian.net/l/c/h13v91c6

 

Salesforce additional  storage procured

Staff report – Shine to send

Powerpoint  - Shine to send

Council approval

 

Security Test Results (PIA/VA/TRA), Security Audit

 

 

Security test results Approval email, related documentation provided by Security ( CISO) Team after security screening the application and integrations.

 

Executive Go/NO GO Readiness Checklist – to be uploaded by Shine prior to GO Live

 

311 Toronto - Consolidated Security Matrix

 

Security Framework

 

Confluence

Salesforce security set up:

https://toronto.atlassian.net/wiki/spaces/EP2/pages/1115357201

 

https://toronto.atlassian.net/wiki/spaces/EP2/pages/1106935907

 

Security matrix link:  311 Toronto - Consolidated Security Matrix

Field level security

 

Email Management

 

 

Email Management

Tableau

 

 

Tableau CRM User Guide - ECRM - Phase II- 311 Toronto CRM Project - Confluence (atlassian.net)

 

https://toronto.atlassian.net/l/c/CxtCr0ob   (This has the recording and documents from the demo on April 28th)

Mobile Application

 

Confluence

https://toronto.atlassian.net/wiki/spaces/EP2/pages/844529721

 

REACT framework?

Mobile app delivery has been postponed to Nov /Dec 2021

KT sessions will be arranged later part of the year.

 

User Acceptance Testing Scripts

 

 

UAT scripts ( Xray link)

 

User Acceptance Testing Results

 

 

Link to test results (Example : A link with all stories in a Sprint with test results attached)

Accessible Testing Scripts

 

 

Testing details, use cases provided by AODA project team

 

Issue navigator - Jira (atlassian.net)

 

Accessible Testing Results(AODA)

 

 

Testing results , Email of AODA testing approval provided by AODA project team

Traceability documentation

 

 

Confluence page to summarize the traceability and mapping for delivered items

Epic > user story > tasks Traceability matrix 

Test case> user story Traceability matrix

 

No Traceability documentation

Approval of Testing results

 

 

UAT Sign off Email 

 

Waiting for final 311 sign off (Oct 08)

Training documentation

 

Confluence

Training:

https://toronto.atlassian.net/wiki/spaces/EP2/pages/2072969270

 

Sustainment handoff:

https://toronto.atlassian.net/wiki/spaces/EP2/pages/2190999553

 

https://toronto.atlassian.net/wiki/spaces/EP2/pages/2196865128

 

https://toronto.atlassian.net/l/c/11z41eW3

 

 https://toronto.atlassian.net/wiki/spaces/EP2/pages/2357133320/Social+Studio+admin+functions.?focusedCommentId=2358444037#comment-2358444037

 

Production deployment plan

 

 

 

311 Toronto CRM - Deployment Plan - ECRM - Phase II- 311 Toronto CRM Project - Confluence (atlassian.net)

10/19/2021   From Catalyst list of all tasks for Prod

Implementation approval documentation (CAB)

 

 

CAB ticket # CM000XXXX

 

Middleware to backend systems connectivity –CAB Oct 13 2021

Production Date(s): 

Oct 19 2021 (Salesforce) -- Preprod

Oct 28' 2021   (P2P1 Integration,   ) 

 

Sustainment Operations documentation (Runbook)

 

 

Runbook has several sections that need to be filled in. These help operational support staff while troubleshooting production issues

 

Confluence page to list items that need to be verified when troubleshooting Production issues (Quick reference)    - Bring this up in the KT session with Catalyst

Exit Warrantee documentation/Approval

 

 

Warrantee Details as agreed to in the SoW etc.

Warrantee start and expiry dates ( 90 calendar days)

 

Oct 30 – Day 1 of warranty >> Jan 2022

Hot fix plan etc. by vendor during warrantee period?

Issue Log

 

 

Issue Log :  TBD , list  to be cleaned up after GO Live

 

Backlog of undelivered items - Outstanding issues after project closes.  TBD , list  to be cleaned up after GO Live

  

Decision Log

 

 

Key Decisions (architecture, Business process, technical process change etc.) taken during project.

File upload solution reviewed with OC office – Email/KIRA task?

Email integration – Email/ JIRA task?

Wordpress instead of Community Portal – Email /JIRA task?

 

JIRA Tasks ( labelled as decisions) – Shine to send link

 

Production software bundle

 

 

Information on any additional software , licenses etc. procured during project delivery

Storage

Lightning Scheduler

Digital Engagement

Social Studio

 

Reference Data Management - Druva Tool

 

Druva –  Application used by Catalyst team to promote reference data between Salesforce environments

3rd party Tool procured by Catalyst on behalf of the City ( 1 year license only, expires ~ Jun 2022, 311/TSD to initiate new procurement if tool needed in future, $4200 ).

KT session on Druva tool:  : https://toronto.atlassian.net/l/c/3U76Y8BY

 

 

Salesforce Shield: _____________________

 

BlueCanvas: __________________________________

 

Correspondence Questions

Contact Management

Configuring Priorities and Scheduled Resolution Dates

Process for Data Load involving Requests

 

Source code of software developed

 

 

Salesforce Change log to include source code change details including pre and post deployment steps.  

Can a report be generated from BlueCanvas for all Salesforce components planned to be promoted to Production?

Mobile App

 

Software tools for sustainment

 

 

Information on any additional software , licenses etc. procured   (example: DevOps tool )

List and describe all software components the system uses, this should also specify sub components of the operating system.

Provide details on purchase, expiry date, licenses etc.

 

Druva – June 2022?  $5000 , 1 org wide license – Catalyst procured it for a year. 311 IBD or TSD?

 

BlueCanvas   - June 2022

 

Software licenses required for sustainment

 

 

Information on any additional software, licenses etc. required for support staff while performing production support activities.

please provide versions and associated licenses to those components

Salesforce: 103 + 20 in p1 , In P2 = 100 SF EE licenses procured

  during launch , Salesforce courtesy licenses = 260

Post launch  Nov 2021 , 123 >> 311 licenses to be transferred from P1 to p2

P2 Lightning scheduler used by 311 booking appointments = 24 Lightning scheduler licenses and 24 restricted user RU licenses

KB = 630 licenses in P2 (215 in p1 >> 630 ) – Also will be used by Divisional users

CTI licenses 80 standard >> 280 Enterprise (p2) – license to be managed by TSD ( Marco?)

Shine initiated process for additional 10  licenses >>  for support team ( Salesforce EE licenses) 

Email to be sent by Shine for above details

 



 

Environments

 

 

Confluence 

Verify below environment information is correct.

 

Also include information on the expected future state of the integrated test environments. Provide information on external Integrations that need to be taken off and connected to another environment etc. after Go Live.

P2 only one integrated end to end environment SIT

Divisional systems /Backend have 2 non prod environments. The test environments that are disconnected from p1 Lagan need to be repointed to P2 UAT. Shine to inform the divisional support teams so the request is in place for the test environments.

 

https://toronto.atlassian.net/wiki/spaces/EP2/pages/1213333505

Environments - ECRM - Phase II- 311 Toronto CRM Project - Confluence (atlassian.net)

 

 

Interfaces to systems operated by other departments

 

 

 

Confluence 

List and describe any interfaces that integrate with other divisional systems.  Documentation required to describe integration details and interfaces of the system.

Departments/Divisions/External using or Interfacing with system:  TW, IBMS, Chameleon?

 

Integrations:

https://toronto.atlassian.net/wiki/spaces/EP2/pages/910655649

 

P1P2 Integration

QA Test cases/scenarios: https://toronto.atlassian.net/wiki/spaces/EP2/pages/2211676165

 

Backup, Restore and Recoverability and Archival of records

 

 

 

Backup, restore and archival strategy for Salesforce application

 

Shine: Not in scope of the project. There is no back up restore solution at an enterprise level

PoC done by Richard Cao- Data Archiver - solution was not approved – Marco and Stephen?

 

Shine to send emails, documentation – on PoC and related decisions

·        Data back-up & Recovery       : https://toronto.atlassian.net/l/c/pYLr5GPK

 

Organization of a Shut-down (Planned Outage) of this application

 

 

 

 

 

What are the steps for a safe outage?

Indicate the anticipated window for applying operating system and application patches.

Indicate the sequenced steps involved in a planned outage for this system.

Indicate whether there would be any damage sustained from an irregular shutdown and whether any repair scripts would need to be run as recourse.

Impact of an outage of an interfacing system on this system

 

Require Technical and Business process?

 

BlueCanvas Roll back : ___________________________

 

 

 

Additional Relevant Documentation for Project

 

 

 

 

Additional Comments

 

 

 

 

For questions – please contact Salesforce Service Delivery Team

 

 

 

 

 

 

Related pages