Bedework Enterprise Calendar System

This has been completely reworked. Making approval of events part of the update caused problems and in any case the actual process of approving or publishing events was wrong. Rather than doing a move the calendar collection path was being updated. It didn’t work for recurring event overrides - probably for a number of releases.

There is now a separate button and action for approval or publishing.

Additionally, it was discovered that moving an event was broken for overrides. This was fixed.

Much of the flow between pages has been updated.

Sharethis - while offering some possibly useful features - definitely has some privacy issues. We ran into this because firefox blocks sharethis at high privacy settings.

This article: https://www.privacyinternational.org/case-study/4403/tracking-service-sharethis-be-profiled provides some interesting info.

Additionally, I tried a single event display with sharethis enabled. The results were:

It probably varies over time but still…​

So for performance and privacy purposes I would suggest sites consider removing sharethis. It has been removed from the xsl.

This release has been a long time coming largely because it involved a significant amount of restructuring. We’ve moved away from ElasticSearch because of licence issues, and completely overhauled the deployment process.

Part of the refactoring is to split off the read-only system from the read-write components. This would allow deployment of a much lighter weight service for feeds and web-presence. This work is not yet complete.

The bulk of the rest of the work is to use jboss modules to deploy all code once only and have that available for all services. This reduces memory usage and startup time.

This release switches from ElasticSearch to OpenSearch due to the licensing issues with ElasticSearch after version 7.10. This will require a reindex of the data but that would be required anyway as we have made too big a jump between versions for an automatic index update to work.

There’s an Amazon perspective on this at https://aws.amazon.com/blogs/opensource/stepping-up-for-a-truly-open-source-elasticsearch/

This has some links to other articles and a search will reveal many others.

While there are reservations about an Amazon supported version it appears to be in their best interests to truly support open source, so - at least for the time being - we have access to a supported Apache 2 licensed search engine.

See Install bedework as a wildfly feature for the new way of installing a working system. This is significantly easier than previously.

This fix is needed for attachments to work correctly. It probably does not affect public events as attachments are not (much?) used. A significant - but long-standing - bug was discovered. Override attachments were not being stored as the table and schema entries were missing. Updating will require adding the table to the database (or recreating the data from the XML dump).

Additionally, ensure the OpenSearch schema is updated (wildfly/standalone/configuration/bedework/opensearch) and reindex after the system is updated.

To fix attachments in postgres.

Ensure all configurations are up to date, especially the OpenSearch schema then reindex the data.

Making a serious effort to get rid of ide warnings. Removing the trivia lets the important stuff stand out. Starting work on implementing new jscalendar and jscontact representations and the jmap protocol.

Changes to log file processor/analyzer. Can produce summary of addresses per ip-address/domain.

There was a long standing bug in category handling for updates. An attempt was made to preserve default categories for calsuites when an event is updated. For example if an event is suggested and accepted the accepting calsuite has its default category added to the event.

This code was being applied to collections which made it impossible to turn off a default category added to, e.g. an alias, by mistake.

This release also introduces a new authenticated public context. This is intended to be used for departmental calendars for example. There were a number of changes needed to make this work but most of the work will come in setting up the calendar collections and aliases. Documentation and examples will follow later as always.

This release mostly consists of upgrades to almost the latest ElasticSearch (always a moving target), the currently latest wildfly and to Java 11 the current LTS release.

Installing the quickstart requires that you first install docker if you wish to have a quickstart image of OpenSearch installed.

There have additionally been some minor changes in configuration and the addition of a tool feature to help in calendar suite creation.

Beyond that there is very little functional change since the last release. However, note that the move to the latest ES required a complete rewrite of the query and indexing modules.

Note: A bug was discovered almost immediately. The commit is at https://github.com/Bedework/bw-calendar-engine/commit/c83e77e3f5ceb990029b84ca7440af83fdc4e568 and a patch:

The Bedework system is divided into two main spaces: the public events space, and the personal and group calendaring space which are kept separate by design. Public events are stored below a public calendar root folder and personal calendars are below a user calendar root folder.

Personal calendar users (and other clients) can subscribe to public events, but users may only add public events using the Administrative and Community Submissions web clients. For more information about Bedework’s public and personal event calendaring models, see Public Events Calendaring and Personal & Group Calendaring.

The following error might be seen when attempting to make changes to any indexed object:

The indexes have been made read-only - this can happen if, for example, the file system becomes full.

The indexes may be unlocked by doing the following

These normally have one or more categories applied for input and output - usually the same category. The categories are applied when Topical Areas are applied when adding or updating events.

When an alias/subscription is selected in the public client (or referenced by a virtual path in feed urls) then the category is used as a filter.

Calendar suites may have subscriptions to the system aliases. Typically, those subscriptions have the same name as the system alias. This added level of indirection helps to ensure consistency in the categorisation.

Or categories if necessary. The normal case is to filter and categorise with the same category.

Go to System → Manage-categories and add any required new category or categories.

Go to System → Manage calendars & folders and determine where in the hierarchy you want the topical area to appear. Usually under aliases.

If you want to group aliases together first create a folder then create the aliases inside that folder

Create a new system folder

Create a new system subscription

You may need to refresh the display to see the new alias.

Using system aliases in this way ensures consistency in categorisation.

Switch to the appropriate group then the Calendar Suite tab.

If the new alias is in a folder it may already be available.

Otherwise

Now go to Main Menu → Add Event and ensure the topical area is visible.

The above needs to be done for the main campus calendar suite. Then

Currently, it appears a restart is needed for it to appear (or a long wait). I’ll try to add a flush to fix this.

Clicking on the "View registrations" button will take you to a screen showing the current registrations for the event. This screen also allows the administrator to hold some tickets and to update or remove registrations.

Administrators may add custom field elements for use during registration. These custom fields are displayed to the user when they register for the event. These fields can be used to obtain extra information from the user as they register, for example dietary restrictions.

Currently the results are only visible in the downloaded registrations.

Data about the event is maintained in x-properties attached to the event and provides the following information:

The simplest data format is standard RFC5545 ics. For greater efficiency the service providing the data should set the Last-Modified header field when the data changes. This will allow the synch engine to skip a download for unchanged data.

Date/time format: Times should beprovided as local times with a timezone - for example

Note that while the standard specifies that a VTIMEZONE component is required for all referenced timezones, this can be ignored. Bedework will use its own set of timezone definitions.

UID: This must be unique and unchanging for each event. It is the primary key to an event - and all its instances if recurring.

Images: Images can be added to the feed and this will allow their display on the main web interface. Specify and x-property as belkow:

Categories: You probably want the incoming events to all be flagged with a specific category or categories. This can be useful for filtering.It makes it easier to include the calendar collections imported by the sync engine. Before carrying out the following steps ensure these categories are created.

For public events there are a few more options available when subscribing.

Process Locations and Contacts: This causes locations and contacts to be moved into x-properties "X-BEDEWORK-LOCATION" and "X-BEDEWORK-CONTACT". The receiving end (bedework) may carry out further actions to validate the location or contact and as a result may set a preexisting location or contact in the event. The x-property is always available for display but this process allows the system to validate the location/contact.

Process categories: This does the same for categories.

Suppress deletion of events? This means that if events disappear from the feed they will stay in the database. This effectively turns the feed into a change feed and can significantly reduce the size of the data. It does mean that events that MUST be deleted will require help from an administrator with sufficient privileges.

As a super user

Locations created and stored within bedework are broken out into a number of fields. Out of these the following are used to create a combined address value which is used for export and import mapping:

These values are separated by a comma and a space. Missing values are skipped.

The update process matches the value of the supplied LOCATION property against that combined value.

If there is an EXACT match then the bedework location will be attached to the syncronized event. Otherwise, the location will be left in an "X-BEDEWORK-LOCATION" x-property. Feeds and the front end display will normally display the x-property as the location for the event.

Contacts are relatively simple and have a single value. If this value matches exactly a defined bedework contact it will be set as the contact value. In all cases an x-property "X-BEDEWORK-CONTACT" will be created

An equivalent process takes place with categories though again, the mapping is much simpler. Categories only have a single value. An exact match in the imported event will cause the bedework category to be applied.

Incoming categories will be placed in "X-BEDEWORK-CATEGORIES" properties.

The use of bedework categories is important for much of the front-end filtering of events as this is driven primarily by categories.

As an example if we have the alias structure
performances/comedy
and comedy is tagged with the category comedy and performances with the category performances then the imported event needs both categories.

Go to the "Users" tab and click on "Manage Admin Roles".

If the user does not appear in the list then type the account into the box labeled "Edit admin roles by userid" and hit enter.

All users should have the "Public Events" flag set.

Users who may create but not publish should have the "Content Admin" flag set.

Users who may create and publish will also have the "Approver" flag set.

Go to the same location, "Users" → "Manage Admin Roles", locate the user in the list and click on the "edit" link.

Flags may be set or unset from there.

Until the user is added to a group or groups they cannot create events. Each calendar suite will have one or more groups associated with it. Typically there is an admin group associated directly with the suite and a submission group which is a member of the admin group.

Approvers are typically added to the admin group and event creators get added to the submission group.

To add a user or group click on "Users" then "Manage Admin Groups".

Find the group to update and click on the "membership" link.

To add a user enter their account in the "Add member" box, ensure "User" is selected then press the "Add" button.

To add a group ensure "Group" is selected.

To remove a group member, click on the trash icon next to the account.

Note that the rights of the users are attached to their auth user entry. So a user is always an approver whatever groups they are in.

There is no provision for a user being an approver in one group and an event creator in another. At some point we may move the rights to the associated groups so that approval rights will come about because of group membership.

When installing bedework on wildfly steps must be taken to ensure that services that should be hidden are not exposed on the network.

Do NOT expose the OpenSearch and database ports by - for example - opening up all ports to the network.

The demo version provides a copy of the minimal opensearch system. This comes with explicit warnings:

The best approach is to disable all ports and only open up those necessary. Bedework only uses the ports 8080, 8081 (and 9990 for the wildfly HAL console). Additionally, even for those ports it is best to pass traffic through haproxy or apache web server and only allow known contexts (/cal, /caladmin etc.) and redirect all unknown traffic to "/cal".

Wildfly can be installed by using the galleon installer, which allows the installation of tailored instances of wildfly.

Additionally extra features can be installed using the same approach. Bedework now can be installed in this way with a choice of features.

This will install a version of wildfly with bedework with a full set of demonstration clients. A demo version of opensearch and apacheds will also be included.

These steps have been tested on a mac and on the latest LTS linux.

These are the steps to install a version closer to a production ready bedework. As noted earlier you should run bedework behind a front-end service such as haproxy or apache. Wildfly can deliver any associated static content so there is no need to move that content elsewhere.

Bedework is installed as a feature pack which has a dependency on wildfly. The galleon tool supports layers which effectively define a particular flavor of the feature.

Many bedework layers have multiple versions for supported databases. For each the appropriate driver will be installed and datasources will be configured. The database is indicated by the suffix "-xx" where xx is:

Following are tables showing all defined layers. Many are flagged as dependencies of others so, for example, bw-calendar-pg will include bw-calendar-ro. In general the only layers to use are the top level layers that specify a database.

The bw-prod* levels provide a more production ready version which will probably still require configuration changes but should be close.

The demo systems will include apacheds as an ldap server with a preconfigured set of accounts - all with the password "bedework". Also the minimal version of OpenSearch will be installed. Startup scripts will be included in the wildfly/bin directory to start and stop each of the services.

Thd next set of layers are used to install specific bedework apps. These could be used to install a specific subset of applications.

The next layers are used to install libraries used by servlet filters for CAS authentication or the keycloak servlet filter for saml V2 (shibboleth).

The next layers are used to install certain functions and may be useful with some of the application layers.

The remainder are dependencies of the other layers..

Note that, while different database layers can be mixed, it’s not clear what will result from selecting the same application for different databases, e.g. bw-public-h2 AND bw-public-pg.

In addition to the bedework layers there are wildfly layers that might be useful.

These assume galleon has been installed and is runnable. For example it may be installed in the home directory and runnable as:

In the following examples we will simply write galleon.sh

This may not work as snapshots can be out-of-date or inconsistent but for reference…​

Download and unzip galleon then run the binary and enter the commands as shown:

The script will ask you if you want the latest, developmentor pre-release version.

You probably want latest. The dev version is obviously for development and can be very unstable. The pre-release version is ther only for testing of the install script.

The script will install apacheds. This will install a directory with some initial ids in place for testing. Ultimately you’ll want to use your own directory service.

You need a running OpenSearch - currently release 1.2.3 The glleon install will provide a script to install a docker image and customize it for the quickstart.

This requires you to have docker and docker-compose installed first.

The instructions related to installing a docker image can be found at https://opensearch.org/docs

After the install is complete use the 2 scripts bwstartosch.sh and bwstoposch.sh to start and stop the docker image.

If you want to install your own copy and configure it the configuration directory is wildfly/standalone/configuration/bedework/opensearch/config. Use this to replace the config in the downloaded OpenSearch.

Maven is required if you want to install the sources. If not you can skip this section.

This is a maven project and as usual you need to set up your maven profile in ~/.m2/settings.xml. The script will display a possible settings.xml file with the paths filled in.

If you want to merge in the profile to an existing settings.xml ensure you also merge in the pluginGroups section.

The profile does not need to be active by default if you have other profiles. The build process will specify the bedework-qs profile.

Below is the contents of an example settings.xml file. This file must be set up before allowing the script to continue on to the builds otherwise they will fail during deployment.

Note there appears to be a bug in the use of activeByDefault. Explicitly specifying a profile with the maven -P parameter is supposed to deactivate a profile marked as activeByDefault. This is not (always?) happening.

If you only have profile marking by default should be fine. To ensure only one profile (or the expected profiles) are active cd into the project and try

The bw and install scripts now source a .bw file in the home directory which can set some defaults.

To alleviate some of the issues of having multiple maven profiles for bedework builds set the profile here with the bw-mvnProfile property. Note the setting needs to include the -P.

When building with maven the profile provides the location of the deployment properties. For the quickstart this is located in the bedework module at bedework/config/wildfly.deploy.properties.

When using the nobuild option you need to take a copy of that file and modify it according to the needs of your installation. Update the .bw file property bw_deployProps to define the location of the modified property file.

An example .bw file is:

During the install an id and password will be set up so that the install process can reindex the data. These will allow use of the cli and the hawtio console.

In many cases it is possible to simply cd in to the appropriate directory and do a mvn install with the bedework-3 profile. However there is a bw script which - while taking longer - does build all projects a module depends upon. This avoids the need to work out the dependency orderings of the independent projects. Thus

will build a lot of projects eventually building the client project which deploys an ear file.

As part of the bedework project there is a maven plugin which uses a properties file to handle post-build deployment issues. Essentially the ear or war as built acts as a template for the deployer.

The deployment process may involve inserting filters for CAS, adding property values to web.xml files, cloning entire wars for calendar suites etc.

The file bedework/config/wildfly.deploy.properties is the quickstart version of that file.

When developing your own service the first thing to do is create a repository with your files and copy the above file into that repository. Then set the org.bedework.deployment.properties value in your maven settings.xml to point to that file.

DO NOT change the org.bedework.deployment.basedir property - unless you move the quickstart. This property is used to locate the wildfly instance.

We’ll describe the process for the main calendar engine. The others are very similar.

All bedework and wildfly configuration files are in standalone/configuration/ - we’ll just refer to the relative path.

TBD

Set the hibernate dialect in the config file:

There are a number of options available. The deployed wildfly demo server has a script to deploy a docker image of OpenSearch. First ensure docker is installed on your system and running then execute:

wildfly/bin/bwinstallosch.sh

To run the image:

wildfly/bin/bwstartosch.sh

will start a container with volumes bound to

This is only suitable in that form for demonstration but may provide the basis for a production service.

For more information see https://opensearch.org/docs

If you’re looking to install as a production service you need to install OpenSearch as a service. There are instructions at the OpenSearch site fo installing on various platforms.

The 2.1.0 release is available at:

After installing and starting the service you will need to restore and reindex the data. For instructions see Reindexing.

If you wish to install OpenSearch as a runnable service there are instructions on running a minimal version without any security. You will need to copy the configuration and possibly the data from the deployed demo wildfly.

In your <home>/.bw file add

Then you can use the restore script to restore the data and indexes:

The public events registration system is enabled by default if you have installed Bedework with the data available in the quickstart.

If you wish to disable the public events registration system remove the "Eventreg admin token" from the System Preferences in the jmx console:

First ensure that the CalWs interface is available. This is the SOAP service that the event registration service uses o communicate with bedework. In system.xml make sure the <calSoapWsURI> element is present and has the same value as the soap:address element in pubcalws-soap/wssvc.wsdl (this requires better explanation - it’s a file deployed in the bw-xml ear)

If you have upgraded from an older release, you may not have the data required for event registration in your system yet. Follow these steps to turn on the event registration system: (Please note: the process outlined below is only set up for the default quickstart and postgresql configs at the moment.)

The event registration service will send notifications to the calendar engine when changes take place that might require notifying users. The event registration service calls the calendar engine notification web service (not to be confused with the notification engine). This web service allows the core engine to add notifications to the accounts of subscribed users. It is the job of the notification engine to forward those via email or some other service.

To configure notifications from eventreg you need to set the BwId, BwToken and BwUri properties in "service=Eventreg"and the NotifierId and NotifierToken values in "org.bedework.bwengine" → "notifications"

[[deploying-sync-engine] === Deploying the sync engine The synch engine handles the synchronization of external subscriptions with a bedework calendar - for example a Google web calendar or an ical feed from a department.

Currently such a synchronization must be carried out to a single calendar collection which only contains data from the external resource. Also only one way synchronization is supported - inbound to bedework.

These subscriptions are available to personal calendar users and to public events administrators. For personal calendar users the options are limited as it is intended only to mirror the external resource.

The synch engine uses an extension of CalWS to communicate with bedework. It requires that the wsdl file contain the location of bedework. This is configured into the deploy.properties file - only one change for the synch engine should be necessary. Set the location of (one of) your application servers in the following.

If you are running everything on one server then the quickstart setting above will do. Note that at the moment the synch engine can only work against a single bedework server. It can accept requests from any member of the cluster however. Keys

Generate a set of keys using the cli.

If you are using multiple servers copy the resulting key file from <quickstart>/wildfly/standalone/data/bedework/ on to each server. Ensure calendar server(s) can locate the synch engine.

The bwengine/synch settings are configured to use a jvm system property to locate the synch engine. In file <quickstart>/wildfly/standalone/configuration/bedework/bwengine/synch.xml you should see:

If you are using multiple servers change the host in <managerUri> to refer to your sync server.

When an event arrives at the receiving end with an "X-BEDEWORK-LOCATION" property if the String value of the x-property is …​ === Locations in bedework

These are stored internally as location entities. They are created withthe admin client for public events or as a result of specifying or receiving locations for user clients.

There is a web service endpoint which can be used to search for public locations. It takes a filter expression as a parameter and will return <a limited?> number

A search takes the form:

The params are * fexpr=expression * ?

The expression is a valid filter expression. Of particular interest are the following

For example

Don’t do this:

Rollback and return an error indication. Should be a bad request - not a 500.

The lastmod db update is not versioned as we get a lot and it causes frequent stale state exceptions.

The update should only take place if the new valueis after the old value.

Also when indexing - if we’re indexing because of a touch - we should ignore version conflict exceptions.

Filter x-properties in public clients/feeds - list of x-props to retain.

Change the api to use response objects throughout. No exceptions. Allows for a better networked api.

Need to be indexed in ES so that finding a notification for an entity is efficient (need to merge multiple notifications for same entity)

Change notifications for public events is probably not working correctly. We should be using the creator - or the owner of the alias - all public events have the same owner (public-user) and change notifications seem to be ending up in that bucket.

For this we would do all interactions with ES and connect to and update db only as needed. Use sequence numbers to ensure db and index correctness. Benefits are shorter db interactions - only at point we update. Less complexity in web clients - no need to have conversations stretching across multiple requests. This can build on the work of the previous item. The web client code is already structured ro assume that it will do an explicit update of entities which should facilitate the change.

Move as much as possible out of the current webapps module into the core APi implementation - this potentially allows a more RESTful style of client - possibly using the new jmap style interface being developed.

Use a larger category scheme as the basis for categories. Use SKOS based representation so that they are RDF friendly. Will allow for sub-categorization by event submitters.

Allow for use of external location sources such as geonames. Will allow good locations on external events.

Search for events near a geo-location (requires locations to have geo)

Finish off the deployment process - it’s THAT close (is there an emoticon for 2 fingers very close together?) to allowing deployers to just replace the ears from prebuilt ears on the site. No builds required - server can detect an update is available.

If all - or many - required dependencies are deployed as wildfly modules it should reduce the size of the deployment and allow for even quicker startup.

Subset of svci but can be used for web client interactions.

Directory interface to directly interact with grouper for user and admin groups. Allow consumer only approach for external management of groups. Use extra ldap attributes to allow admin groups to be maintained in ldap.

Use kibana to get metrics etc

Update UI to provide a search - possibly based on map. Use tzdist geo feature (being developed)

Currently I’m indexing all of the instances we allow - that can be many. The ideal would be not to index any instances but only the master and overrides. This has some problems.

Time range searches are easy. We have an indexed start and end and for the master we set it to be the entire (perhaps infinite) range so it always gets included in the result. For this case we could generate the instances based on the master.

For a query involving other conditions - e.g. categories the master may not appear.

To run some basic tests there is a script bw-caldavTest/src/main/resources/calconnect-tester/testbw/bw-QuickLook-CalDAV.sh

This script sets up the tester which needs to be cloned from the repository.

Running the tester requires that a number of users be provisioned in a particular state.

The quickstart data comes with users user01-user04. Each is setup with

(Replace 01 with 02-04 for the rest)

January 9, 2010
* Public calendaring New default public theme based on Duke/Yale themes Feed URL and Widget Builder for generating rss, json, xml, and ical feeds as well as embeddable javascript widgets A "feeder" application that serves as a common source for public data feeds and widgets A web cache application for storing and serving the feeds and widgets Internationalized and modularized themes New mobile theme for iPhones and other smartphones