This file contains notes about the status and progress of TBWiki

---

Contents: Next major items to work on Things to do (random ideas, un-triaged) Major Features that are not complete enough to use User accounts Table usability BugList, FIXTHIS, Regressions Table of combined issues Regression count Other items to work on, by category Style General Security Processors Blog ProgressChart SlideShow ImageGallery Tables Forms editing and Templates Users and attributes Long term ideas Things Completed

Note that some test materials are on a separate wiki at: http://bird.org/cgi-bin/test.cgi/FrontPage

Before working on this wiki any more, re-read this article by a developer who has written 7 personal wikis:

• https://borretti.me/article/unbundling-tools-for-thought
  • the basic idea is that there are better tools for thought than personal wikis
    • but that's OK - tbwiki is a hobby project that is fun to work on

Next major items to work on [edit section]

• support captions
  • if caption="foo bar" is an attribute of an image, use figure, and figcaption to surround the image.
  • see https://www.w3schools.com/TAGS/tryit.asp?filename=tryhtml_figcaption_css

• MacroCategory.py - show a list of pages in a particular category
  • implement category tags, like this:

     {{Category(Tasklist,2023)}}
     {{Category(New Feature)}}
    

  • implement category reference, like this:

     {{ShowCategory(Tasklist)}}
    or
     {{Category(show:Tasklist)}}
    

• Blog processor
  • support adding a blog entry
• (done) ExportSite processor
• Search macro:

   {{Search(word or phrase)}}
  

• do performance enhancements, possibly using the Scalene profiler
• add a "blog" processor
  • for ideas, see: https://jekyllrb.com/docs/posts/
    • save blog pages as: blog-YYYY-MM-DD
    • don't support comments
    • show index of blog posts, current post
  • research blog layouts:
    • see Blog Layouts and Features for ideas
• figure out why I keep getting logged out (is there something wrong with the cookie expiration date?)
• don't add an extra line when saving.
  • use splitlines instead of split to avoid adding an extra line on an edit??
    • there appears to be no split here
  • do some testing to see where the extra \n gets introduced:
    • client side or server side?
    • in form population (send from server) or in form response data (sent from client)?
  • is there a way to detect a new newline, and remove it?
  • grep split tbwiki_engine.py | grep [\\]n
• get css from page area of local wiki
  • don't retrieve from /var/www/css
• Support wiki on mobile devices
  • (done) fix up mobile theme
    • change mobile.css and mobile.footer and mobile.header
    • some pages still show very small fonts on my phone
  • autodetect mobile phone browser, and use mobile theme automatically
    • user UserAgent field
• switch to wsgi - I considered flask, see FlaskResearch
  • should switch to wsgi - I have experience with that
• enhance blog system (see TBWikiToDo#Blog)
• add page lock (to support multi-user)
• add help routine to all processors
  • add SystemInfo processor to handle help requests
    • with help action item, that calls help with a processor name
    • processors only have to return a block of tbwiki markup
• make tables robust
  • eliminate save_table_data - placing support in appropriate routines in the io_handler (tb.handler)
    • one of write() or some other class method.
  • fix bug 017. See BugList
  • fix bug with edit-in-place on QuarterlyData table
  • make automated test suite for testing page rendering
  • fix all known bugs, with all known table types
    • make a table test matrix
• table flow control
  • for transcription program (ProcessorTranscriber.py)
    • make a view form and edit form for the transcriber database
      • read documentation on view and edit forms
    • add processor_name to the transcriber database config
• add support for hybrid tables
  • use case is to add a spreadsheet formula to the FIXTHIS pages, to show totals
  • ability to have tables with records from different sources
    • add io_handler per-record
    • need permission checks (readable, writable) per-record

Things to do (random ideas, un-triaged) [edit section]

• don't allow caching a page edit page
  • use http meta cache-control directive to prevent caching
  • set caching relatively low anyway
• add page locking for edits - to prevent collisions
  • lock a page for 15 minutes when an edit is started
    • create a lock file in the admin directory
      • put user, start time, and page hash in lock file
    • start a timer to handle lock timeout
  • generate a hash of the page when the edit starts
  • if someone else tries to edit, give them a warning that they have lower priority - indicate who has the lock
  • when someone submits a page, compare the page hash and lock
    • if page hash doesn't match, don't save the data
    • remove the lock
  • after 15 minutes, remove the lock
• support arbitrary field addressability in json data:
  • JsonPageFile:field.list_field[n].foo.bar
• fix bug: edit-in-place wipes out conf entries in inline block

• add 'tbclient list' feature
  • modify tbclient to send 'list' request
  • modify ProcessorDataHandler.py to respond to list request
• add TODAY() function to spreadsheet

• add websed over snapshotted values
  • add websed over diff of snapshotted values
• add chart over snapshotted values
  • use plot charts?
• add access-controlled content
  • the purpose of this is to add a page with contact information for other test projects, on the fuego wiki
  • see https://moinmo.in/HelpOnAccessControlLists
  • at the block level, with ProcessorAccessControl
  • at the page level, with accesscontrol directive
  • put required permissions at top of block
  • prevent reading or writing to block (or page)

• add link to page for file column in attrdb and parsed data files (to go to page)
  • This is handy for FIXTHIS entries, and for editing
• _bug0013 - can't edit-in-place the buglist

Major Features that are not complete enough to use [edit section]

User accounts [edit section]

• Creating and administering user accounts
  • need to re-test user accounts, with new features (next_num)
  • Users can create accounts, but physical access to the server is required to change admin flags, or to remove an account
    • may be able to use a special version of attrdb (raw page edit) for user accounts. It wouldn't be great but would get the job done for now.
    • is the user edit form not working??

Table usability [edit section]

• Specific table usability features:
  • other usability feature is deciding which links to use for a table (I never use query, and always have to decide between 'edit-in-place' or 'edit')

BugList, FIXTHIS, Regressions [edit section]

• See the BugList page for bugs that need to be worked on.
  • that bug list is from a database. Here is one that just text on a page:
    • Manual Bug List
• See the CodeFixthisList page for a list of FIXTHIS items in the code
• See the PageFixthisList page for a list of FIXTHIS items in the pages

Table of combined issues [edit section]

Date  ^	bug_count  ^	FIXTHIS_count  ^	missing_FEATURES  ^	total  ^
2017-03-08	15	xx	xx	15 + xx + xx
2016-08-12	14	94	2	110
2016-08-02	13	92	3	108
2011-04-21	17	78	1	96
2008-10-06	17	78	1	96
2008-02-04	13	66 (55 cgi-bin+ 11 plugin)	3	82
2007-07-25	9	62	3?	74
2007-02-06	9	41	5	55
2006-11-15	8	35	6	49
2006-10-06	8	37	7	52
2006-08-25	6	35	5	46

Query

---

Regression count [edit section]

Periodically, check the regression test page to see if things still work as expected.

Here's a table, manually maintained, with regression testing history.

date  ^	version  ^	commit  ^	OK  ^	missing output  ^	bad  ^	bad page list  ^
2016-08-12	1.6.2	a45cce77	91	2	2	TestMarkupVariables, TestMarkupVariablesDyamicTest
2016-07-18	1.5.3	61eb2f1e	39	31	4	TestMarkupVariablesStatic, TestSandbox, tbwiki.footer, tbwiki.header

Query

Other items to work on, by category [edit section]

Style [edit section]

• enhance themes feature
  • see DocThemes
    • for an example of self-editability - see http://kernelnewbies.org/navigation_menu
  • calculate theme from user account or other variable
    • current use of page name is lame
  • allow configurable navigation links
  • allow configurable toolbox links
  • read page dressing from nested pages??
    • allow header and footer pages to be HTML or TBWIKI format
      • if TBWIKI, allow for sub-page references that are loaded, parsed and formatted recursively
• tweak mobile.css
  • see w3c pages about automatic layout - http://www.w3schools.com/html/html_layout.asp
• switch to MediaWiki page syntax?? (not yet)
  • see http://meta.wikimedia.org/wiki/Help:Wikitext
  • need to switch bullets handling
  • need to switch table parsing
• have css come from page area??? (would allow editing css using wiki)

General [edit section]

• convert upload to be a processor
  • allows putting upload on any page
  • control number of files via config
  • can specify link(" action=upload&dest_filename=foo") for missing files - to prompt user if file is missing
  • upload result should be shown in page message?
• add page rename
  • can do create, move content, delete, but that's awkward
• generate tbwiki stats from log
• cache page content, for static content
  • change all prints to output routines (to control routing of output)
• allow administrators to view and edit files in the 'admin' directory using normal tbwiki access (just like regular pages)
  • treat it like a MoinMoin overlay directory
• make a mobile viewer
• add ability to grab content at specific intervals
  • kind of a combination of crontab and websed
  • allows for charting something automatically over time
  • make a crontab processor
    • present a form for when to run, and action
• make a syntax for block names that are not included in block output
  • we have anonymous blocks, hidden blocks, and named blocks
    • named blocks show both the name and the content. It would be nice to hide the name
  • possibly: #.BlockName ??
• make tbwiki more relocatable (for use on laptop)
  • on bird.org:
    • move *.cgi scripts from www/cgi-bin to ~/work/tbwiki/cgi-bin
    • put link in www/cgi-bin
    • add base_dir to *.cgi scripts
  • write script for laptop that does:
    • downloads latest tbwiki and tbwiki-data
    • adjust base_dir in *.cgi files
    • adjust base_url in *.cgi files
    • adjust plugin symlinks in tbwiki-data dirs
• make unit test system - see Unit Test Notes
  • NOTE - already have ProcessorRegressionCheck.py
  • have a page specify a test
  • have markup, and html and compare them
    • here's one unit test:

             record=bold 1
             markup='''bold'''
             html=<b>bold</b>
      

  • make a ProcessorUnitTest.py processor
    • read all "UnitTest.*" pages, convert markup to html, compare to html
    • report results
• put this categorized todo list in a table?
• change main() in tbwiki_engine.py
  • call main when tbwiki_engine.py is called from command line
  • make a tbwiki.py command?
    • program should have the ability to create a static page from the command line
      • e.g. tbwiki.py FrontPage -o FrontPage.html
    • program should have the ability to run unit tests?

Security [edit section]

• check permissions on table write operations
• allow administrators to view and edit files in the 'admin' directory using normal tbwiki access (just like regular pages)
  • treat it like a MoinMoin overlay directory
• have rights affect the displayed table edit links

Processors [edit section]

• add help routines for all Processors
• add a processor for "select an item"
  • used for choosing a group of words to practice
  • used for choosing a story to read in scriptures
  • present a list of items from a database, that are unselected
    • allow a user to select one
  • or, select an item randomly from a list of database records
    • allow user to select it or keep choosing a different one
  • have 3 categories: unselected, in-progress, finished
    • database must "state" field?
  • processor allows only legal state transitions
    • specify legal state transitions (or is this too meta?)
• support mapping routines (e.g. to convert chapters to pages and back)
• ProcessorScriptureLookup.py - maybe should be a macro?
  • take a reference, and replace it with the text for the versus
  • this should be very handy for church talks, lessons, etc.

Blog [edit section]

• add right sidebar with:
  • recent posts
    • categories
    • search bar
    • tags
• use individual pages for each post
  • create add-post form
    • page schema has title, content, date, tags, id
• navigation to previous and next posts
  • response area
    • show threaded responses with "reply" button on each one
    • response form with "post comment button
• show date, tags, response count
• see example: https://lava-block.com/godot-engine-tutorial-making-a-2d-game-ui-mechanics-sound/
• support customizable background image
• ability to drag and drop images to upload to wiki

ProgressChart [edit section]

• add support for goal lines to the ProgressChart processor
• handle chart attributes
  • make chart_attrs_class, to generate defaults when attrs are missing
  • create flat namespace for chart attributes
  • translate chart atttributes into call arguments during chart creation
• un-specialize pychartwrapper.py
  • chart another data type using same wrapper??
    • generic multi-series data chart
    • scripture reading chart?
    • weight chart
    • running chart??
• support data entry and modification for TimsRunning data
  • see ProcessorProgressChart.py
  • use Add Form for table??
    • support current date for date entry??

SlideShow [edit section]

• SlideShow
  • make sure all nested parsing works
  • add sidebar list of slides (section 1 headings), with links to pages
  • support show attributes and per-slide attributes (bgcolor, bgimage, margins, etc.)

ImageGallery [edit section]

• ImageGallery
  • support links=1 - to show individual images in full screen
  • autodetect image size for full-screen (standalone) image gallery
    • use Javascript to detect browser window size??
  • support num_images - to show only some of the images
    • gallery has prev and next links on sides (similar to blog navigation)
  • support image borders in ProcessorImageGallery
  • support link to fullsize image in ProcessorImageGallery
  • support displaying the filenames in ProcessorImageGallery
  • add javascript zoom functions

Tables [edit section]

• add sortby=select_order - to specify that text fields should be sorted in the order that they appear inm the select possible_values list.
  • this helps the user control the sort order for things like word-based priority: ex: very high, high, medium, low, very low
• add messages to table module
  • allow for outputting messages about an individual table
  • table.add_table_message(msg)
  • this would be handy for table-specific error messages
  • should have block-specific messages in general
• allow for configuration of table control links
  • have user specify which links to show, and where
• use better filenames
  • use unique part of filename for record_id
  • allow user to pick filename when creating the record
• support edit-in-place for tbwikidb files
  • use get_section and set_section
    • modify to read/write definition lines
• eliminate difference between record, record_id
• use tbwikidb or attrdb for user database??
  • this puts human readable filler on every page, but that's the wiki way
  • this allows creating a multi-file database just by creating the template
  • this is fragile - human editors will break the pages parseability
• add table query cheat sheet to query table form page
• make prototype family info database on bird.org
  • automatically create a User account file on login???
  • make MemberXX records for family members
    • determine the schema (what fields)
    • create a table view with contact information
• table queries and views FEATURE
  • support <table>RecordView
• support read-only attribute on whole table
• Support time-sampled database
  • schedule a websed operation that grabs data at crontab-like intervals
    • or that creates a new record whenever the page is accessed (and schedule something to access the page automatically at periodic intervals)
• support automatic data snapshotting
• write a 'table creation wizard' processor
  • ask for field names and types, and create the schema file and cover file (or template file, if its a tbwikidb)
• document table config concepts:
  • base config (from original conf block)
  • dynamic config (from request form variables)
  • query config (from query conf block)
• saved queries
  • specify query list in conf block
  • each item in list specifies a query conf block
• make backup file when editing table files
• finish all routines for editing a table
  • use {{{Table(arg1,arg2,arg3...) notation for inline tables
    • support other args:
      • formspec, query line (row filter, column filter, etc.)
• document Table use with data_scan module
  • DocTables has some information (one example, the FIXTHIS table)
• multi-file tables
  • display multi-line values correctly
  • display '<no-data>' values correctly
  • auto-sense type of each field (text, select, textarea) and use for auto-form generation
• handle field types with enumerated values (lists and radio buttons)
• document table architecture and interfaces - see Table Architecture
  • document table attributes
  • document chart attributes
• make ProcessorTablePick.py
  • See Table Pick Page
  • present simplified list of "unpicked" items
  • allow user to pick one item
  • save picked item with attribute "state" changed to "in-progress"
  • present detailed list of "in-progress" items
  • allow user to mark an item as "done"
    • save done item with attribute "state" changed to "done"
  • whether user wants to have multiple items in "in-progress" state is their choice
• queries, query_lists

Forms editing and Templates [edit section]

• use form emitter for user accounts.

Users and attributes [edit section]

• Support online creation of new users
  • make a "create user" form
  • create a new user account file from a form
• provide help if user forgets password
  • link to e-mail the password??
  • password hint??
• create 'user.edit' action, which edits a user account file
• support different roles (admin vs. non-admin)
• allow user to edit their attribute values
  • make an "edit user" form
• REFACTOR - use existing form code for user forms
• make user account management it's own processor module
  • ProcessorUser.py - we now have sub-actions, so this should work.
    • it needs to access table operations, but in a special way

Long term ideas [edit section]

• add 'save expected output and show again' to RegressionTest processor
• add link to backup page on history page
• look at other wikis for feature ideas
  • in particular, ikwiki statically compiles the wiki on every page edit
• documentation
• table editing:
  • should support "save and edit again" and "save and add another row" buttons for saving edit forms.
• Table Of Contents
  • do auto-table of contents, like wikipedia??
  • handle included files and text expanded from processors??!!
• check HTML and CSS on multiple browsers with BrowserCam or browsershots
  • see http://browsershots.org/http://bird.org/tbwiki/FrontPage
• make a journal processor (based on progress chart)
• make a blog processor (with threaded comments)
• make a forum processor (with threaded and voted-on comments)
• make a page cache
  • add dependencies to a makefile, and rebuild cache when dependencies change
• put config into admin page?
  • reduce cgi-bin to just some directory references

Things Completed [edit section]

• 2021-03 Remove '?action=save' from url in edit form
• 2018-10 Add image rotate to ImageGallery app.
• 2017-04 add triggerd - ability to detect triggers that are due and execute them
• 2017-03 add ProcessorTrigger - ability to trigger a snapshot
• 2017-03 add ProcessorSnapshot - page/block/table snapshot function
• 2017-02 add tbclient - ability to get,put and edit items in the wiki
• 2017-01 add page trails
• 2017-01 add relative index to cell address (so you can do $.-1)
• 2016-12 ProcessorImageGallery - finish showing images in full size page
• 2106-11 add ability to customize table operation links (edit_links, sort_links)
• 2016-9 add ability to read a specific block, by number
• 2016-08 convert to per-table messages (for debugging and errors)
• 2016-08 support editing sections with nested sub-sections
• 2016-08 allow default_values with generated values (%(timestamp), %(next_int), %(next_page_num))
• 2016-08 add page delete
• 2016-07 add tbwikidb database type
• 2016-07 fix data_scan module to parse multi-line fields
• 2016-05 make ProcessorImageGallery
• 2016-05 make ProcessorBlog
• 2016-03 make better diff output
• 2016-03 put engine_version in data_class, and add to footer
• 2016-03 add "printable format" option for page
• 2016-03 made a test wiki at: http://bird.org/cgi-bin/test.cgi/FrontPage
• 2014-05 make talk pages optional - see config.has_talk_page global config
• 2010-10 support inline tables - FEATURE
• 2010-?? websed - add websed feature to scrape other web pages
  • 2010-?? data-scan - add as a processor first, then make into a database grabber
• 2008-10 table - back up page on table edit
• 2008-09 chart - draw chart of table data
• 2008-08 processor - add a charting package (pychart)
• 2008-02 table - added support for sorting table rows
• 2008-02 table - support col_list and row_filter specifiers
• 2008-02 table - added ability to edit whole table at a time
• 2008-02 macro - create Include macro
• 2008-02 table - separated conf into separate class
  • add field_db (parsing, handling)
    • add field types, colors, possible values
  • add routine to read block from page
• 2008-02 table data - separated IO handler into separate class
• move tbwiki with authentication to bird.org
• aug03 - add user accounts.
  • put user account files in an 'admin' directory
  • authenticate user from files in 'admin' directory in action user.login
  • look up user by name to process login form (in action user.login)
• jul07 - add cookie code for user accounts
• feb07 - use pages for page dressing (see tbwiki.header and tbwiki.footer)
• jan07 - support file upload FEATURE
• 11/15 - add support for inline images
• change backup routine to save current version of page instead of old version of file
• save edit comment and show in history
• edit sections FEATURE
• parse and format:
  • bullets
  • headers
  • horizontal rules
  • paragraph breaks
  • preformatting
• write edit form for page
• save edited data to file
• read page from data directory and emit as html
• write handlers for various actions
• create add_record form (using existing edit_form code)
• write mtable.save_form() routine
• re-write edit_form to present whole table with edit links
• support Table internal macro, with args
• modified backup time in file extension to be human-readable
• wrote mtable.modify_record() routine
• wrote mtable.remove_record() routine
• parse and display URLs as links
• use pages for page dressin
• make items in table of contents into links
  • make an anchor target for each heading
  • use href for anchor target in TableOfContents macro
• make nicer format for table of contents box
• add search form to header
• support multi-page table source_spec
  • support e.g. "User*" as the spec
    • read all pages starting with User
      • finish parse_table_pages() routine
        • read each page matching the source_spec
        • parse out values using a page template
          • start with simple name=value parser
    • create page on add record
    • write page on edit record
    • remove page on remove record
• support history action
  • show history from backup times
  • allow diff between two versions