Sakai Feeds

This is an abstract I prepared for the 2012 Blackboard Developer Conference – BbDevCon. We shall see if it gets accepted. IMS Learning Tools Interoperability 1.0 now has very broad market adoption and has been in Blackboard since Release 9.1SP4. Blackboard has added LTI support to building blocks, making it very simple to add LTI to a building block. Developers can plug externally hosted learning tools into Blackboard and the rest of the marketplace with a few lines of PHP. Now that the low-level LTI “plumbing” is in place, what will we do with it. This talk looks at the tools that are available in the marketplace that support IMS LTI and show them plugged into Blackboard. We will introduce and describe IMS Learning Tools Interoperability 1.1 that includes support for returning grades from external tools back to the grade book and demonstrate this. This talk also looks at ways to quickly build and host tools that function as LTI Providers and plug those tools into Blackboard. This talk also looks at the next release of IMS Learning Tools Interoperability 2.0 that includes even simpler provisioning and installation of tools, expanded grade/outcome services, and improved ability to import and export classes with links to dynamic content and services hosted on the web.

By Phil HillAlong with others, I have written several times over the past 12 months here, here, here and here about the significant changes occurring in the educational LMS market. In my opinion, when we look back on market changes, 2011 will stand out as the year when the LMS market passed the point of no return and changed forever. What we are now seeing are some real signs of what the future market will look like, and the actual definition of the market is changing. We are going from an enterprise LMS market to a learning platform market. What I mean by ‘enterprise LMS’ is the legacy model of the LMS as a smaller, academically-facing version of the ERP. This model was based on monolithic, full-featured software systems that could be hosted on-site or by a managed hosting provider. A ‘learning platform’, by contrast, does not contain all the features in itself and is based on cloud computing – multi-tenant, software as a service (SaaS). The 2011 EDUCAUSE event captured the zeitgeist of the changes, as it seemed most of the buzz at the conference centered on new LMS solutions and paradigm changes. Instructure made their debut at the conference, Pearson’s OpenClass was announced, Blackboard announced a new move in open content focused on CourseSites, and Cengage demonstrated their MindTap platform. Rather than slowing since EDUCAUSE, we have seen several additional announcements in the past three months. CourseKit was released as a free learning platform targeted at faculty adoption. Apple’s iTunesU app was announced alongside the iBooks / Author textbook offering, extending iTunesU as an iPad-based learning platform. Facebook made a move within its higher education roots, starting a pilot program with Groups for Universities. In my post from last summer, I characterized the changes we were starting to see, but with all of the recent changes, I think it would be useful to extend the first two trends mentioned. The question is, what will the LMS market that is emerging from these changes look like?  No one can know for sure what will happen over the next 3 – 5 years, but I do think there are some key trends that are worth understanding. The market is more competitive, with more options, than it has been for years.  Instructure is a real player that has shown that it can win against established LMS vendors with big wins in Utah and at Auburn.  LoudCloud has new clients at CEC, Grand Canyon U and an unreported win at a public state university.  BrainHoney won at BYU.  Pearson LearningStudio has major wins at Arizona State and Columbia online programs.  Desire2Learn has roughly doubled in size in the past year.  Moodle and Sakai, including through providers such as MoodleRooms and rSmart and Unicon, continue their impressive wins in the market. In terms of market competitiveness, we are seeing even more offerings than mentioned in August, including a new class of “free”. Pearson’s OpenClass, Blackboard’s CourseSites, CourseKit, Apple’s iTunesU app, and Facebook’s Groups all join NIXTY as free learning platforms. We have not had the time to see the market share changes based on these new offerings, but if nothing else, there are even more choices now. Related to the above, there is a trend towards software as a service (SaaS) models for new LMS solutions.  The SaaS model offers some compelling advantages in terms of deployment time and ability to mine and report transactional data that might not be possible with other approaches.  SaaS is not a panacea, but this is a growing trend in the LMS market. The trend towards SaaS could perhaps more accurately be described as the default model now for new offerings. In the LMS market from just short two years ago, the default model was enterprise LMS. The only exception was Pearson’s LearningStudio (the artist formerly known as eCollege.com). Today, every single new offering mentioned above is SaaS-based. Apple’s iTunesU app is a mobile app, but the content is served from a behind-the-scenes SaaS platform. Perhaps more significantly – there has not been a new enterprise LMS created since around 2004. Yes, each legacy LMS provider has major new releases available, but the one exception you could argue is that Sakai 3 is a new LMS and not just an upgrade from Sakai 2. Other than this exception, every new LMS solution to enter the market in the past two years has been based on a learning platform. I doubt we will see any more enterprise LMS solutions created given the cost-benefits of creating SaaS offerings. Another trend that is becoming apparent is that many of the new offerings are not attempting to fully replace the legacy LMS, at least all at once. Rather than competing with all of the possible features that are typical in enterprise LMS solutions, the new platforms appear to target specific institutional problems and offer only the features needed. Perhaps inspired by Apple’s success in offering elegant solutions at the expense of offering all the features, or perhaps inspired by Clayton Christensen’s disruptive innovation model, the new learning platform providers are perfectly willing to say ‘no – we just don’t offer this feature or that feature’. My colleague Jim Ritchey has written about the changes that SaaS models are starting to have in the higher education ERP market, put in context of the Datatel+SGHE merger. His key point: Therefore the challenge for the vendors is how to get the ERP, with its slow development and implementation cycles, to provide the solutions to the new needs of the institution. In the LMS market, the new answer to this question – how to adapt and respond to new institutional needs – appears to be based on learning platforms. Possibly related posts: What Platform Do You Use for (Pure) Distance Learning? I’m doing a little research and could use your help.... Oracle's New Academic Enterprise White Paper The product group I’m in at Oracle (Academic Enterprise Solutions,... Zimbra: What a Mashup-Enabled Enterprise App Looks Like Phew. Enough with the Apple stuff. I actually still have... Enterprise vs. Internet World Views in Educational Tool Design There’s an excellent (albeit necessarily technical) conversation about implementing OKI... Sakai Foundation Board Platform: Vision for the Technology I am honored to announce that I have been nominated... Farewell to the Enterprise LMS, Greetings to the Learning Platform by %%AUTHORINK%% on e-Literate

Before that scares the hell out of anyone using Solr, the Solr bundle I am talking about is a small shim OSGi bundle that takes content from a Social Content Repository system called Sparse Map and indexes the content using Solr, either embedded  or as a remote Solr cluster. The Solr used is a snapshot from the current 4.0 development branch of Solr. Right, now thats cleared up I suspect 90% of the readers will leave the page to go and read something else ? So Solr 4 works just great. The applications using Sparse Map, like Sakai OAE , have a high update rate and are adding to the index continuously. The bundle queues updates and processes them via a single threaded queue reader into the index which is configured to accept soft commits and perform periodic flushes to disk. The Solr instance is unmodified from the standard Solr 4 snapshot and we have had no problems with it. Provided the cadinality of the fields that the application indexes are not insane, and the queries are also achievable there are no performance issues with queries being the sub 10ms that we have all become accustomed to from Solr. Obviously if you do stupid things you can make a query in Solr take seconds. There are however some issues with the way the bundle works and certainly when deployed into production into a real cluster there are issues. No one would seriously run the Sparse Map with this Solr bundle on a single app server node for anything other than development or testing, so the default Embedded Solr configuration is a distraction. If your not writing code with the intention of deploying into production, then why write the code? Life is to short, unless your an academic on track to a Nobel prize. When deployed, the bundle connects to a remote Solr master for indexing with one or more Solr slaves hanging off the master (polling not being pushed to). There are several problems with this configuration. If the master goes down, no index updates can happen. This doesn’t break the Solr bundle since it queues and recovers from master failure with a local write ahead transaction log or queue. It does break the indexed data on the master since anything in memory on the master will be lost, and only those segments on disk will get propagated to the Solr slaves when the master recovers. This is a rock and a hard place. 1s commits with propagation cause unsustainable segment traffic with high segment merge activity. Infrequent commits will just loose data and destroy data propagation rates. The slaves, being read only are expendable provided there are always enough to service the load. Thats sounds like the definition of a slave, I would not like to be one, but then I wouldn’t know if I was. Solr, in this configuration, wasn’t really designed for this type of load. If we indexing new documents at the rate of 1 batch an hour then Solr in this configuration would be prefect. However the updates can come through at thousands per second. So although it works, its fine, but when it breaks it will break and leave the index in some unknown state. The problem is rooted in how the indexing is done and where the write ahead log or queue is stored. Its fine for a single instance since the write ahead log is local to the embeded Solr instance but no good for a cluster. Other approaches There are lots of ways to solve this problem. It was solved in Sakai 2 (CLE) search which treated segments as immutable and sent them to a central location for distribution to each app server. Writers on each app server wrote to local indexes and on commit the segment was pushed to a central location where the segment was pushed to all other app server nodes. The implementation was less than perfect and there were all sorts of timing issues especially when it came to merging and optimising. That code was written in 2006 on a very old version of Lucene (1.9.1 IIRC). So old it didn’t have commit, let alone soft commits and it was only used for relatively slow rates of update supporting non critical user functionality. Its in production many Sakai 2 schools. Every now and again a segment gets corrupted and that corruption propagates slowly over the whole cluster with each merge and optimise. Eventually full index rebuilds are needed which can be carried out when in full production but are best done overnight when the levels of concurrency are lower. At the time we had considered using the DB based IndexReaders and IndexWriters from the Compass project. These were readers and writers that used a DB BLOB as the backing store. Lucene depends heavily on seek performance, and doing seek over a network into the DB blob, doesn’t work. The IO required to retrieve sections of the segments to pull terms is so high that search speed is a bit low (British understatement, stiff upper lip and all that). After tests those drivers were rejected for the Sakai 2 work. It might have worked on an Oracle DB where seeks in blobs is supported and you can do some local caching, but on MySQL it was a non stater. The next approach is that used by Jackrabbit. The Lucene index is embedded in the repo. Every repo has a local index with updates being written directly to all index sychronised across the cluster. Works well on one app node, but suffers in a cluster since ever modification to the local index has to be serialised over the entire cluster. Depending on the implementation of that synchronisation it can make the whole cluster serialized on update. Thats ok if the use case is mostly read as it is with the Enterprise Content Management use case, but in a Social Content Repository the use case is much higher update. App servers cant wait in a queue to get a lock on the virtual cluster wide index before making their insert and inserting a pointer into a list to tell all others their done. Since 2006 the world has not stood still and there have been lots of people looking at this space. LinkedIn opensources Zoie and Bobo that deliver batched updated into distributed indexes and then build faceted search from those indexes. Although these would work for a Social Content Repository my feeling was the quality of data service (time it takes from a content item update to the index presence) was too high and required lots of discipline in the coding of the application to ensure that data local to the user was published directly to the content system rather than discovered via the search index. The area of immediate impact of data for LinkedIn is well defined, the users view of their profile etc so that QoDS can be higher than where an update might have to instantly propagate to 100s of users. The types of use cases I was targetting with the Sparse were more like Google+ where groups take a greater prominence. Except in Education, the group interaction is real time which pushed the QoDS down into the second or sub second range. So Zoie was ground breaking, but not enough. The work on this application, now Sakai OAE, started in 2008 when there was nothing else (visible) around. We started with SLing based on Jackrabbit and use its search capabilities, until we realised that a Social Content Repository has to support wide shallow hierarchies with high levels of concurrent update the Enterprise Content Management model is deep narrow hierarchies with lower levels of concurrent update. See this for detail Roll forwards to 2010 when we pulled in Solr 4 which was just about to get the NRT patches applied. It looked, bar the small issue of cluster reliability that it was an Ok choice. And now were up to date 2012 and the world of distributed search has moved on and I want to solve the major blocker of reliability. I don’t want to have to write a distributed index as I did for Sakai 2, partly because there are many others out there doing the same thing better than I have time to. I could use SolrCloud, although IIUC that deals with the cloud deployment of Shards of SolrSlaves rather than addressing the reliability of high volume updates to those shards. Terms, Documents or Segments What to shard and replicate. The ability to shard will ensure scalability in the index, which turns the throughput model from a task compute farm into a parallel machine using the simplest of gather scatter algorithms (my PhD and early research was numerical parallel solutions on early MPP hardware, we always looked down on gather scatter since if never worked for highly interconnected and dynamic problem sets, sorry if thats offensive to MapReduce aficionados, btw gather scatter is the right algorithm here). The ability to replicate, many times, will ensure that we don’t have to thing about making hardware resilient. But what to shard and replicate. The Compass IndexReader and IndexWriter DB implementation proved that inverted indexes need high seek speeds to minimise the cost of scanning segments for terms. Putting latency between the workings of the inverted index and its storage was always going to slow an index down and even if you made segment and terms local to processing, processing queries on partial documents (shards of terms) creates imbalance in the processing load of a parallel machine and dependence on the queries. The reason for less than perfect parallel speedup on numerical problems in 1990 was almost always due to imperfect load balance in the algorithm. Pausing the solution for a moment to wait for other machines to finish is a simple bottleneck. Even if sharding and replication of partial documents or terms balances over the cluster of search machines, the IO to perform anything but the simplest query is probably going to dominate. So I need an index implementation that shards and replicates documents. Its 2012 and a lot has happend. The author of Compass Shay Banon (@kimchy) went on to write ElasticSearch with a bunch of other veterans. It looks stable and has considerable uptake with drivers for most languages. It abandons the store segments centrally model of Compass and Sakai 2 and replicates the indexing operation so that documents are shaded and replicated. Transporting a segment over the network after a merge operation, as Solr Master/Slave does is time consuming, especially if you have everything in a single core and you merged segment set have become many GB in size. This looks like a prime contender for replacing the search capability since its simple to run, self configuring and discovering and ticks all the boxes as far as scaling, reliability and ease of use. Another contender is Lucandra. Initially this was just Lucene on top of Cassandara. It implemented the IndexReader and IndexWriter inside Cassandra without segments eliminating the need to manage segments but also loosing most of the optimisations of memory mapped data. Unlike the Compass IndexReader and IndexWriter that wrote segments to DB blobs the structure of the index is columns and rows inside Cassandra. Not dissimular from the Sparse Map Cassandra driver that indexes by writing its own inverted index as it goes. There are some performance gains since if you put the Lucandra class into the Cassandra JVM the data is supposedly local, however Cassandra data is replicated and shaded so there is still significant IO between the nodes and the solution may benefit from Cassandras ability to cache, but will still suffer from the same problems that all term based or partial document sharding suffers from. Poor performance due to IO. When Lucandra became Solandra a year later in the authors reported the performance issues, but also reported a switch to sharding by document. There will be more out there, but these examples show that the close source community implementing large distributed indexes on a document based shard and replicate approach is the right one to follow. (Hmm isn’t that what the 1998 paper from some upstarts titled “The Anatomy of a Large-Scale HypertextualWeb Search Engine” said ? The authors of Solandra admit that it still looses many of the optimisations of the segment but rightly point out if your deploying infrastructure to manage millions of small independent indexes then the file system storage issue become problematic which is where the management of storage by Cassandra becomes an advantage. As of September 2011 I get the impression that ElasticSearch is more mature than Solandra, and although everyone itches these days to play with a new tool in production (like a column DB) and throw away the old and reliable file system, I am not convinced that I want to move just yet. Old and reliable is good, sexy and new always gets me into trouble. I think, I am going to deprecate the Solr bundle used for indexing content in Sparse Map and write a new bundle targeting ElasticSearch. It will be simpler, since I can use the write ahead transaction log already inside elastic search, its already real time (1s latency to commits and faster than that for non flushed indexes). I have also found references to it supporting bitmap bloom filter fields which means I can now embed much larger scale ACL reader indexing within the index itself. A post to follow on that later. Watch this space.

The Australian National University has launched myANU, the staff and student portal, based on uPortal 3.2.4 from Jasig. https://my.anu.edu.au Complementing this launch, ANU are also releasing the ANU Secure Login service, based on the Central Authentication Service (CAS) 3.4.10, also from Jasig. I have been the architect, evangelist and lead developer on both of these projects since late 2009. As anyone in the University community knows, the wheels turn slowly, however it is a fantastic day to have both of these projects finally out the door! Congratulations also to my development team, David Clarke and Osama Alkadi and previous developers Denny and Jiang Chao as well as the testers and BA’s involved. And not forgetting all of those people in the community I have bugged on email and IM! myANU, together with CAS, will pave the way for a truly integrated online experience for staff and students of the Australian National University, well into the future. More posts to follow about what services we are offering through the portal. Stay tuned!

I am very pleased to announce that the Basic LTI Portlet has had an update and is released as version 1.3.0. This is a portlet that implements the IMS Basic Learning Tools Interoperability specification and allows you to render any Basic LTI enabled application inside uPortal (or any other JSR-168 compliant portlet container). Possibilities include Sakai tools, Peoplesoft components, tools from other LMS’s, collaboration and learning tools, blogs, forums, wikis, the list is endless. This release includes additional adapters for connecting to Noteflight and Chemvantage, an improved Basic LTI launch when using uPortal, as well as some other minor UI improvements. This portlet is running in production in the Australia National University‘s portal, my.anu.edu.au, based on uPortal 3.2.4. For downloads, configuration, screenshots and all other information, head to the Jasig wiki: https://wiki.jasig.org/display/PLT/Basic+LTI+Portlet

I am very pleased to announce that the Sakai connector portlet has had an update and is released as version 1.4.0. This portlet allows you to render tools from your Sakai environment inside uPortal. It is completely configurable, and each user can choose a tool from any of their Sakai sites to display in the portal. It uses a combination of web services and and implementation of the IMS Basic Learning Tools Interoperability specification to connect you to Sakai from within uPortal. This release includes an improved Basic LTI launch, as well as an improved editing UI for when multiple connector portlets are on screen at once. This portlet is running in production in the Australia National University‘s portal, my.anu.edu.au, based on uPortal 3.2.4. For downloads, configuration, screenshots and all other information, head to the Jasig wiki: https://wiki.jasig.org/display/PLT/Sakai+connector+portlet

Profile2 1.4.6 has been released. This is a minor feature release for the 1.4 series adding the ability to disable certain aspects of the profile depending on institutional requirements. It also adds the ability to set a user’s profile image based on their user type. This allows for better multi tenancy support – just give a user a different user type and then they can get a default image for that institution. The notes for this release are available on Jira: https://jira.sakaiproject.org/browse/PRFL/fixforversion/12590 As always, everything is configurable, so check the settings: https://confluence.sakaiproject.org/display/PROFILE/Profile2 Upgrading is as simple as bumping the version in master/pom.xml and rebuilding (then removing the old jars from shared/lib). If you are upgrading from a release prior to 1.4.5 you will need to check for any required database conversions: https://source.sakaiproject.org/svn/profile2/tags/profile2-1.4.6/docs/ Happy networking!

The brilliant folks at Lancaster University have re-written the Sakai CLE Roster as a Javascript/HTML application (using trimpath) and it has now been released as 1.0.0. Roster2 has the same feature set, permissions and UI as Roster classique, but with a much improved integration with Profile2. There are also a number of settings you can use to configure it. Available as a binary (indie release) and as a source tag. https://confluence.sakaiproject.org/display/RSTR/Roster2

By Audrey WattersThe learning management system upstart Instructure is unveiling Canvas K-12 today, a version of its platform aimed — as the name suggests — for the K-12 level. The company says that it’s already had over a dozen school districts adopt Canvas, even before this roll-out of a specially designed LMS. Traditionally the LMS has been something implemented primarily by colleges and universities, but as more and more K-12 schools move to online learning and digital curriculum, there’s a growing demand at that level. It’s a hot market, and according to research published in December 2011 by Simba Information, “the LMS segment is expected to grow at a compound annual rate of 7.3%, reaching $377 million by the 2014-2015 school year.” As such, it’s hardly surprising to see some of the big education companies make their move to offer schools these services. The acquisition of Edline by Blackboard last fall made it clear that the learning management giant was serious about its push into that market. But as the Simba research suggests, it’s a market that’s still up for grabs. While Blackboard still holds a little over half of the higher ed LMS market, Blackboard, Pearson and Moodle altogether share only about 30% of the K-12 market. That provides an interesting opportunity for Instructure, which officially launched its cloud-based LMS this time last year. Its new K-12 offering includes several new features aimed at this level: it contains Common Core standards and objectives so that it’s easy to align assignments with them. There are also analytics for districts, schools, teachers and parents to be able to assess student progress. And that parent piece is particularly important as parents will have access to their child’s information, and just as importantly, have access to Instructure’s messaging system — so you can get an SMS when your child doesn’t turn in a homework assignment or an email with the week’s spelling list and so on. Despite competition from some of the big LMS players, Instructure has made some inroads into higher education. Can it do the same at the K-12 level? When I spoke to CEO and founder Josh Coates yesterday, he noted that the company’s recent trip to FETC made them realize that a lot of K-12 teachers are fairly unfamiliar with the idea of what an LMS even is. (That’s something that should make us ask if an LMS is even necessary.) Of course, Instructure isn’t selling to teachers (although it does offer a free product that any teacher can adopt). It’s selling to districts. But that the LMS is a new(ish) thing to the K-12 level might just work in Instructure’s favor, even if the startup remains a relative unknown. If schools choose to adopt an LMS because of their move online, then a Web-friendly, user-friendly, cloud-based tool (with easy Google Apps for Edu integration) might just fit the bill. That is, if the price is right, something that makes the future of that K-12 market — what with shrinking K-12 budgets and options for free and low-cost alternatives (“apps” not “systems”) — more than a little uncertain. Possibly related posts: Sakai 3: What It Is and When To Move To It I have been getting a lot of questions from the... ANGEL's Open Source Move ANGEL Learning has announced that they have incorporated TiddlyWiki into... What Makes It Great? When I was in college, I was very fortunate to... Instructure and Security Testing Instructure has had a very interesting reaction to the news... Instructure Canvas: A New LMS Entrant We’re making progress on getting the Sakai conference keynote videos... Instructure Makes Its Move into the K-12 Market by %%AUTHORINK%% on e-Literate