Attendees

  • Michael Drake
  • Vincent Sanders
  • Daniel Silverstone

Apologies

  • John-Mark Bell

Outstanding work (from May)

  • Monkey
    • Driver - capability to "Click" on a specified piece of text (Daniel) (Basically a way to say click button FOO) DONE
  • Framebuffer
    • Language support for resources (Vince)
    • Listing of compiled-in surfaces (Vince)

Topics

Discussions

Frontends

Coverity

We went over coverity output because we got upgraded. Fixed a few items along the way.

Bug Triage

Plenty of triage was done. In addition, we fixed:

Activity

Michael

  • Coverity fix for LibCSS that makes the select hash insertion at start of non-empty list code more readable.
  • Worked with Daniel on fixing NULL node hover crash on page with all content set to visibility: hidden.
  • Much plotting and planning of libnslayout's text layout, with Daniel.
  • Looked at browser window scale stuff with Vince and moved the scale handling for invalidate to the core, and updated the GTK front end.
  • Setting up of frameworks for new libparagraph stuff.
  • Updated duktape to version 2.4.0.
  • Made local-history bitmaps scale with DPI
  • Made iframe test on https://test.netsurf-browser.org/html/iframe.html
  • Attempted to add support for core window scroll_visible to RISC OS front end.
    • It didn't compile, so built the RISC OS toolchain and SDK, and then fixed the compilation error.
  • Added support for scaling keyboard shortcuts to the Framebuffer front end.
  • Optimised libnsfb's xrgb8888 surface rendering using a neat trick from Adrian Lees.
  • Added support for scrolling core windows to the RISC OS front end.
  • Fixed the RISC OS core window get_window_dimensions to handle toolbar height.
  • Made all the core window callbacks return nserror instead of void.
    • And updated all the front ends.
  • Changed the core window interface for scrolling from scroll_visible to much simpler get/set scroll APIs.
  • Reimplemented old scroll_visisble API in the core on top of the new set/get scroll and get_window_dimensions, using Daniel's algorithm from the GTK front end.
  • Made the RISC OS local history window claim input focus when it opens.
  • Fixed a couple of Duktape bindings for DOM element nodes that didn't check the return value of a LibDOM call.
  • Suppressed dead store scan-build warnings in the CSS presentational hints gatherer.
  • Wrote some simple iframe and image tests for bad SSL.
  • Implemented the libparagraph stuff for creating and destroying contexts.
  • Implemented the libparagraph stuff for adding content to contexts.
  • Added the libparagraph stuff to the Makefile, to build it.
  • Looked into crash with animated GIF redraw.
    • Optimised box_coords when there's a float in the ancestors.
    • Potentially fixed crash by short-circuiting object redraws before the HTML has had a layout pass.
  • Looked at CSS styling stuff with Vince.

Daniel

  • A number of bug fixes
  • Worked with Michael planning text layout
  • Acquired a polyfill for Array.from() and fixed consoleFormatter.
  • Fixed a typo (BUTTOM)
  • Made it so that cURL fetches will abort early if possible
  • Updated stacktrace handling to new duktape
  • Added cookie setting support
  • Wrote reload support into monkey driver/farmer
  • Added a cookie test using JS and a new cookies CGI
  • Various cleanups with fetch_fdset() vs schedule_run()
  • Added extra diagnostics during monkeyfarmer shutdown
  • Fixed the oldest bug! (Added scroll-visible to local history and added key navigation)
  • Did similarly to treeview (scroll-visible)
  • Added click support to monkey, monkeyfarmer, monkey-driver, and added a test to netsurf-test which will use it.
  • More robustification of the asyncio loop in monkeyfarmer.
  • Added additional intuitive nav to local history
  • Added a 'aborted-fetch' test (and fixed monkey_driver a bit)
  • Migrated the callback for auth/ssl from llcache through hlcache to the users of the hlcache. It should be functionally equivalent (except sub-fetches no longer prompt but instead automatically fail like in other browsers).
  • Designed an approach for shadow contents, fleshed it out, discussed it among ourselves, discarded as being overcomplex. It is preserved below for posterity.
  • Designed, with Vince primarily being the clever monkey, a new approach which removed the query callbacks from the llcache and instead uses content generated in the about: handler. We think this is cleaner.
  • Combined CONTENT_MSG_ERROR and CONTENT_MSG_ERRORCODE
  • Taught browser_window how to remember fetch parameters so that we can restart fetches in the new query design. Split navigation in half to make that clean.
  • Migrated SSL certificate chain storage to browser_window and allowed it to store the chains for the completed fetch, leading to possibilities of padlocks later down the line. In the process, fixed a long-standing bug where serial numbers never worked.
  • Excised the llcache query pathways and migrated them all the way up to the browser_window
  • Tracked down a strange behaviour and discovered we didn't handle PAUSED coming out when we tried to finish the parse. Added a fix for that.
  • Added internal navigation concept to browser_window and migrated the login handling inside it, using Vince's new auth query support in the about: fetcher.

Vince

  • Restored OpenBSD to functionality in the CI
  • Splitting up of browser.c
  • Removed scale from _invalidate.
  • Removed scaling from get_dimensions.
  • Re-fixed Haiku worker in CI.
  • Cleared out leftover junk builds from the CI carrier.
  • Tested Michael's RISC OS changes under RPCEmu.
  • Completed scaleectomy in browser_window_mouse_click(), browser_window_mouse_track(), browser_window_get_features() browser_window_set_scale(). And updated all of the callers and all of the front ends.
    • Now the front ends don't need to care about scale.
  • wrote about scheme query handlers for auth and privacy

Thoughts on shadow contents -- We decided that this was overcomplex and CANCELLED THIS IDEA

In order to complete the work we started my migrating queryies all the way up from the llcache to the browser_window, we need to come up with a way for the browser_window to load and display a real content which will represent the query to present to the user.

Terms

  • real content: A real content if available (e.g. the current page when then navigating away).
  • loading content: The content which is currently loading into the window.
  • real loading content: The loading content which may trigger a query.
  • query: The callback from the real loading content which needs user input to decide on the action for.
  • stored query: A stored callback and private word for answering the last query which came for the real loading content.
  • shadow loading content: A holding space for a real loading content which should not give any events unless something else unblocks the query in another window.
  • special navigation: An attempt to navigate in such a way as it would answer a stored query

Events

The events we have to respond to are:

  1. The query event coming into the browser_window_callback
  2. Any attempt to browser_window_navigate
  3. Any attempt to browser_window_reload
  4. Any event coming from a shadow loading content
  5. Destroying a browser_window
  6. Any call to browser_window_stop

States

The states we might be in are:

  1. Not loading: The normal state of being where we're not loading and as such nothing interesting is going on.
  2. Normal loading: The process of loading is happening, there is no shadow loading content and the real loading content is making progress.
  3. Shadow loading: The process of loading is paused, and a shadow page is being loaded. This implies there's a stored query.
  4. Shadow loaded: This is like the not loading state, but there is a shadow loading content and a stored query which mark it as different.

State transitions

  1. Moving out of not loading happens as normal on any of the usual events
  2. When normal loading all the usual events behave in the current manner, except if we receive a query. On receipt of a query the real loading content is moved from the loading content into the shadow loading content, and a new request is created in the loading content for the query page with requisite parameters. Finally the query parameters are stored into the stored query. This moves us to shadow loading.
  3. When shadow loading and any of the navigation events occur, we first abort the loading content and move the shadow loading content (which is the real loading content) back into the loading content. Then we refuse the stored query which will abort the real loading content. Then we process the navigation as before.
  4. When loading completes when we're in shadow loading then we proceed as "normal" to replace the current content with the converted shadow document, we fire the query at the frontend, and we enter shadow loaded.
  5. When in shadow loaded and normal navigation occurs, we behave essentially like we would in shadow loading and such an event occurred.
  6. When in shadow loaded and we receive a special navigation instruction, then we gather the information from the URL to answer the stored query, then close the query page content, move the shadow loading content back to the loading content, and then clear the stored query. This transitions us back into normal loading. Finally we send the answer to the query which will kick off normal fetch procedures once more.
  7. When in shadow loaded or shadow loading we receive any events from the shadow loading content we should assume that a decision was made in another window and so we abort any loading content, or close any displayed content, drop the stored query, put the shadow loading content back into the loading content, and return to normal loading

Replacement query proposal

In order to simplify matters we decided instead that we would unwind the paused/callback mechanism and instead have the fetch stop with an appropriate error code indicating the query, allow the browser window to deal with that and then to restart the fetch from the top level.

To do so, the browser window will have to retain the post data etc. This allows for us to also implement the when reloading a page which was posted, query the user for whether or not to re-post. (but this is 'some future time')

Then when we receive a query form CONTENT_MSG_ERROR the browser window can issue a special internal fetch for the query page it needs, passing the requisite information. On CONTENT_COMPLETE it can then call the gui, giving it a chance to chime in.

If a reload happens while we're on a special page, we redo the original fetch, if we navigate to the special try-again, we redo the original fetch. If we navigate to the special proceed, we perform the update as needed, and redo the original fetch.

Step one

First, we need to clean up and unify CONTENT_MSG_ERROR and CONTENT_MSG_ERRORCODE.

This will allow us to later use this unified thing to communicate more useful fetch errors upward toward the browser window.

Step two

We make browser_window_navigate store the fetch parameters so that redoing the fetch is possible

Step three

We change the fetch/llcache/hlcache layers to always pass SSL certificate chains upward when they are available.

And we change the browser_window to store that chain alongside both the current_content and the loading_content.

We may add APIs to access these later.

Step four

We change the fetch/llcache/hlcache layers to remove the QUERY concept entirely and to pass an appropriately structed CONTENT_MSG_ERROR.

We change the browser_window to consume that error and call the current callbacks. The proceed callback will have to be implemented in browser_window in a simplistic fashion.

At this point, all the plumbing is in place, but the current behaviour in terms of GUI interaction and urldb remains the same.

Step five

We write a handler in about: to provide the query pages. This is MVP to be able to get on with things. At this point we allow browser_window to have the concept of a special fetch which does not update the fetch paramters, nor the url bar etc. And when we stop a fetch because of the query errors, we navigate to that about: handler using the special fetch mode, thus we'll end up with both the new page and the GUI prompt still. Ideally we move the gui prompt to be on CONTENT_DONE for the special load.

Step six

We handle the special about: URI which means 'proceed' and the one which means 'try-again' as per the descriptions in the summary of this proposal.

Step seven

Rework the gui/browser_window querying so that instead of proceed/cancel it is try-again/take-answer/proceed.

Step eight

Various sundry cleanups as a result of all of the above.

Statement of work

If at all possible, we'd like to see some of the following addressed before the next developer weekend…

  • General
    • Remove 401 login implementations from all frontends (Vince) DONE
    • Resolve issues arising from introducing internal query pages (Daniel) DONE
    • Implement any appropriate auto fill auth handlers
    • Low level cache should own certificate data and persist with all other data
      • Shouldn't be pushed down to browser window; browser window should request it from the hlcache handle, which should get it from the llcache.
    • Implement Site Information (padlock) and repurpose certificate error core windows to be certificate viewing interfaces
    • Continue styling the generated query pages.
    • Review TODOs.
  • Framebuffer
    • Language support for resources (Vince)
    • Listing of compiled-in surfaces (Vince)
  • Internationalisation
    • Write the English query page page messages first. (Vince) DONE
    • Make a mailing list post about query pages. (Daniel) DONE
    • Translations of all messages for the SSL/privacy internal query page.
    • Translations of all messages for the authentication internal query page.
  • Current CI tests (test-site-run)
    • Three groups:
    • Tests that only fetch from our infrastructure.
    • Tests that fetch off third party servers and are fast to run.
    • Test that fetch from third party servers and long running.
    • In Jenkins: Group the first two and make them trigger the third one.
  • New tests
    • Take the 10,000 sites list, split it up into blocks of 20, and run them as independent tests.
    • Four core VM, which gets 4 Jenkins jobs for running these tests.
    • Experiment and see how it breaks down. Needs to be weekly (we can tweak it).
  • Text layout
    • Continue implementing. (Michael)

Next time

  • Cambridge.
  • 30 Nov / 1 Dec.
  • Consider extending either side, depending on holidays.