In 2.3.19, sharing cache storage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

In 2.3.19, sharing cache storage

Jaime Garza-2
Hi Daniel,

I am trying to share an instance of MruCacheStorage across threads. I keep it around across multiple threads and when creating my Configuration, I simply do a Configuration#setCacheStorage(). This internally calls Configuration#createTemplateCache, which calls cache.setConfiguration(this), which calls TemplateCache#clear(), which eventually clears the storage.

My use case is that I know that a set of around 200 threads will be accessing the same templates over and over (executing the same template(s) on a different root namespace) concurrently. I want the loading of templates to happen as infrequently as possible. Also, since I am on a slightly customized version of 2.3.19, I cannot come to the greatest new version.

The question (or questions) is: is there a way to implement my strategy? My current workaround is nasty. I have created a subclass of MruCacheStorage, which reads a flag "canClear" from thread local, and then decides whether to let the clear proceed or not. I sandwitched the call to setCacheStorage() with canClear = false and canClear = true

        SharedCacheSettings.disallowClear();
        config.setCacheStorage(storage);
        SharedCacheSettings.allowClear();

AND

public class SharedCacheStorage extends MruCacheStorage {
    public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
        super(maxStrongSize, maxSoftSize);
    }
   
    @Override
    public void clear() {
        if (!SharedCacheSettings.canClear())
            return;
       
        super.clear();
    }
}

What is your opinion about my hack? What kind of trouble am I inviting?

Thanks in advance.
Jaime Garza

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: In 2.3.19, sharing cache storage

Daniel Dekany
The problem with the approach is that Template-s are bound to a
Configuration instance (they store a reference to it), and sometimes
may reach back to it during template execution. However, can't you
just use a common Configuration instance on all those places that
share templates?

(Out of curiosity... what's the modification in 2.3.19 is about?)

--
Thanks,
 Daniel Dekany

Wednesday, July 23, 2014, 5:51:52 PM, Jaime Garza wrote:

> Hi Daniel,
>
> I am trying to share an instance of MruCacheStorage across threads.
> I keep it around across multiple threads and when creating my
> Configuration, I simply do a Configuration#setCacheStorage(). This
> internally calls Configuration#createTemplateCache, which calls
> cache.setConfiguration(this), which calls TemplateCache#clear(),
> which eventually clears the storage.
>
> My use case is that I know that a set of around 200 threads will be
> accessing the same templates over and over (executing the same
> template(s) on a different root namespace) concurrently. I want the
> loading of templates to happen as infrequently as possible. Also,
> since I am on a slightly customized version of 2.3.19, I cannot come to the greatest new version.
>
> The question (or questions) is: is there a way to implement my
> strategy? My current workaround is nasty. I have created a subclass
> of MruCacheStorage, which reads a flag "canClear" from thread local,
> and then decides whether to let the clear proceed or not. I
> sandwitched the call to setCacheStorage() with canClear = false and canClear = true
>
>         SharedCacheSettings.disallowClear();
>         config.setCacheStorage(storage);
>         SharedCacheSettings.allowClear();
>
> AND
>
> public class SharedCacheStorage extends MruCacheStorage {
>     public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
>         super(maxStrongSize, maxSoftSize);
>     }
>    
>     @Override
>     public void clear() {
>         if (!SharedCacheSettings.canClear())
>             return;
>        
>         super.clear();
>     }
> }
>
> What is your opinion about my hack? What kind of trouble am I inviting?
>
> Thanks in advance.
> Jaime Garza


------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: In 2.3.19, sharing cache storage

Jaime Garza-2
Oh, using a shared configuration would be acceptable? That is great to hear. That is what I have now, but I thought that would be an anti-pattern. I will continue with that.

The modifications that I have done have to do with adaptations to FM to make it more of a DSL. We use FM for formatting emails and SMS messages. I have modified mostly FMParser.jj keeping a series of svn patch files for each one of my modifications, hopefully helping me move to a more recent FM. We have extensions for sql queries with a <#data> directive, data phrasing as a builtin for sequences, ?unique for sequences, etc. We maintain our own FM document based on the 2.3.19 docs. We needed some validation support, multiple error messages per template (one error was not acceptable,) and others.

Thanks
Jaime

-----Original Message-----
From: Daniel Dekany [mailto:[hidden email]]
Sent: Wednesday, July 23, 2014 11:22 AM
To: Jaime Garza
Cc: [hidden email]
Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage

The problem with the approach is that Template-s are bound to a Configuration instance (they store a reference to it), and sometimes may reach back to it during template execution. However, can't you just use a common Configuration instance on all those places that share templates?

(Out of curiosity... what's the modification in 2.3.19 is about?)

--
Thanks,
 Daniel Dekany

Wednesday, July 23, 2014, 5:51:52 PM, Jaime Garza wrote:

> Hi Daniel,
>
> I am trying to share an instance of MruCacheStorage across threads.
> I keep it around across multiple threads and when creating my
> Configuration, I simply do a Configuration#setCacheStorage(). This
> internally calls Configuration#createTemplateCache, which calls
> cache.setConfiguration(this), which calls TemplateCache#clear(), which
> eventually clears the storage.
>
> My use case is that I know that a set of around 200 threads will be
> accessing the same templates over and over (executing the same
> template(s) on a different root namespace) concurrently. I want the
> loading of templates to happen as infrequently as possible. Also,
> since I am on a slightly customized version of 2.3.19, I cannot come to the greatest new version.
>
> The question (or questions) is: is there a way to implement my
> strategy? My current workaround is nasty. I have created a subclass of
> MruCacheStorage, which reads a flag "canClear" from thread local, and
> then decides whether to let the clear proceed or not. I sandwitched
> the call to setCacheStorage() with canClear = false and canClear =
> true
>
>         SharedCacheSettings.disallowClear();
>         config.setCacheStorage(storage);
>         SharedCacheSettings.allowClear();
>
> AND
>
> public class SharedCacheStorage extends MruCacheStorage {
>     public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
>         super(maxStrongSize, maxSoftSize);
>     }
>    
>     @Override
>     public void clear() {
>         if (!SharedCacheSettings.canClear())
>             return;
>        
>         super.clear();
>     }
> }
>
> What is your opinion about my hack? What kind of trouble am I inviting?
>
> Thanks in advance.
> Jaime Garza


------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: In 2.3.19, sharing cache storage

Daniel Dekany
Wednesday, July 23, 2014, 8:48:28 PM, Jaime Garza wrote:

> Oh, using a shared configuration would be acceptable? That is great
> to hear. That is what I have now, but I thought that would be an
> anti-pattern. I will continue with that.

That's how the guide and the javadocs shows it as well. The
configuration meant be a singleton, used from multiple threads. (I
often see the mistake where people re-create the Configuration for
each request. I don't know where that comes from, but that can totally
kill performance.) Of course, then you have independently developed
components in the same JVM that use FreeMarker internally, they will
have their own singleton configurations, and they should have, but
those won't use common templates anyway.

> The modifications that I have done have to do with adaptations to
> FM to make it more of a DSL. We use FM for formatting emails and SMS
> messages. I have modified mostly FMParser.jj keeping a series of svn
> patch files for each one of my modifications, hopefully helping me
> move to a more recent FM. We have extensions for sql queries with a
> <#data> directive, data phrasing as a builtin for sequences, ?unique
> for sequences, etc.

You can have custom directives and functions without modifying FM, as
you certainly know. Surely they are not as flexible/terse as what can
be achieved with core directives and built-ins... Making them closer
is a future goal.

> We maintain our own FM document based on the
> 2.3.19 docs. We needed some validation support,

Validation of what?

> multiple error messages per template (one error was not acceptable,)

Is that multiple parsing errors or?

> and others.

--
Thanks,
 Daniel Dekany


> Thanks
> Jaime
>
> -----Original Message-----
> From: Daniel Dekany [mailto:[hidden email]]
> Sent: Wednesday, July 23, 2014 11:22 AM
> To: Jaime Garza
> Cc: [hidden email]
> Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>
> The problem with the approach is that Template-s are bound to a
> Configuration instance (they store a reference to it), and sometimes
> may reach back to it during template execution. However, can't you
> just use a common Configuration instance on all those places that share templates?
>
> (Out of curiosity... what's the modification in 2.3.19 is about?)
>
> --
> Thanks,
>  Daniel Dekany
>
> Wednesday, July 23, 2014, 5:51:52 PM, Jaime Garza wrote:
>
>> Hi Daniel,
>>
>> I am trying to share an instance of MruCacheStorage across threads.
>> I keep it around across multiple threads and when creating my
>> Configuration, I simply do a Configuration#setCacheStorage(). This
>> internally calls Configuration#createTemplateCache, which calls
>> cache.setConfiguration(this), which calls TemplateCache#clear(), which
>> eventually clears the storage.
>>
>> My use case is that I know that a set of around 200 threads will be
>> accessing the same templates over and over (executing the same
>> template(s) on a different root namespace) concurrently. I want the
>> loading of templates to happen as infrequently as possible. Also,
>> since I am on a slightly customized version of 2.3.19, I cannot come to the greatest new version.
>>
>> The question (or questions) is: is there a way to implement my
>> strategy? My current workaround is nasty. I have created a subclass of
>> MruCacheStorage, which reads a flag "canClear" from thread local, and
>> then decides whether to let the clear proceed or not. I sandwitched
>> the call to setCacheStorage() with canClear = false and canClear =
>> true
>>
>>         SharedCacheSettings.disallowClear();
>>         config.setCacheStorage(storage);
>>         SharedCacheSettings.allowClear();
>>
>> AND
>>
>> public class SharedCacheStorage extends MruCacheStorage {
>>     public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
>>         super(maxStrongSize, maxSoftSize);
>>     }
>>    
>>     @Override
>>     public void clear() {
>>         if (!SharedCacheSettings.canClear())
>>             return;
>>        
>>         super.clear();
>>     }
>> }
>>
>> What is your opinion about my hack? What kind of trouble am I inviting?
>>
>> Thanks in advance.
>> Jaime Garza


------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: In 2.3.19, sharing cache storage

Jaime Garza-2


> -----Original Message-----
> From: Daniel Dekany [mailto:[hidden email]]
> Sent: Wednesday, July 23, 2014 1:06 PM
> To: Jaime Garza
> Cc: Daniel Dekany
> Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>
> Wednesday, July 23, 2014, 8:48:28 PM, Jaime Garza wrote:
>
> > The modifications that I have done have to do with adaptations to FM
> > to make it more of a DSL. We use FM for formatting emails and SMS
> > messages. I have modified mostly FMParser.jj keeping a series of svn
> > patch files for each one of my modifications, hopefully helping me
> > move to a more recent FM. We have extensions for sql queries with a
> > <#data> directive, data phrasing as a builtin for sequences, ?unique
> > for sequences, etc.
>
Yeah I know that. We have a myriad of functions we implement. Our additions look, as you mention, terse. It appears as part of the language, which it now is. It would have been a dream to be able to add additional built-ins, and I have been following the list for a while now, and I know that this is a common ask and that you are thinking on ways to do it. I look forward to the moment we can have "legal" built-ins in the environment. Adding directives in a standard way with <# would have been ideal as well.

> You can have custom directives and functions without modifying FM, as you
> certainly know. Surely they are not as flexible/terse as what can be achieved
> with core directives and built-ins... Making them closer is a future goal.
>
> > We maintain our own FM document based on the
> > 2.3.19 docs. We needed some validation support,
>
> Validation of what?

Users declare their own namespaces in the UI. These are attached to database rows and appear in the root namespace. We need to know if they are really used in the templates at all, for backend optimizations. We warn users if they declare them and then they do not use them. For this purpose we follow templates depth-first in a parsing phase as much as possible. If the include uses string expressions, we warn too, as we cannot proceed. We use the parse flag to follow includes. Etc. It is kind of elaborate, but it ended being very powerful.

In fact, I have been following the discussion of editor plugins and how we can get the parsing tree, and how people have done this in their own way, and then communicate with you. This is a way in which we discover the namespace for us, in-depth. I instrumented FMParser's PrimaryExpression and Include for this. I did not have to open DollarVariable.java or Include.java for it.

>
> > multiple error messages per template (one error was not acceptable,)
>
> Is that multiple parsing errors or?

We do multiple parse errors, and lexical errors as well. We implemented a recover mechanism to stabilize the parsing process as much as possible, by skipping tokens, and backtracking. We then return a list of errors as opposed to one error. This was a big change.

>
> > and others.
>
> --
> Thanks,
>  Daniel Dekany
>
>
> > Thanks
> > Jaime
> >
> > -----Original Message-----
> > From: Daniel Dekany [mailto:[hidden email]]
> > Sent: Wednesday, July 23, 2014 11:22 AM
> > To: Jaime Garza
> > Cc: [hidden email]
> > Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
> >
> > The problem with the approach is that Template-s are bound to a
> > Configuration instance (they store a reference to it), and sometimes
> > may reach back to it during template execution. However, can't you
> > just use a common Configuration instance on all those places that share
> templates?
> >
> > (Out of curiosity... what's the modification in 2.3.19 is about?)
> >
> > --
> > Thanks,
> >  Daniel Dekany
> >
> > Wednesday, July 23, 2014, 5:51:52 PM, Jaime Garza wrote:
> >
> >> Hi Daniel,
> >>
> >> I am trying to share an instance of MruCacheStorage across threads.
> >> I keep it around across multiple threads and when creating my
> >> Configuration, I simply do a Configuration#setCacheStorage(). This
> >> internally calls Configuration#createTemplateCache, which calls
> >> cache.setConfiguration(this), which calls TemplateCache#clear(), which
> >> eventually clears the storage.
> >>
> >> My use case is that I know that a set of around 200 threads will be
> >> accessing the same templates over and over (executing the same
> >> template(s) on a different root namespace) concurrently. I want the
> >> loading of templates to happen as infrequently as possible. Also,
> >> since I am on a slightly customized version of 2.3.19, I cannot come to the
> greatest new version.
> >>
> >> The question (or questions) is: is there a way to implement my
> >> strategy? My current workaround is nasty. I have created a subclass of
> >> MruCacheStorage, which reads a flag "canClear" from thread local, and
> >> then decides whether to let the clear proceed or not. I sandwitched
> >> the call to setCacheStorage() with canClear = false and canClear =
> >> true
> >>
> >>         SharedCacheSettings.disallowClear();
> >>         config.setCacheStorage(storage);
> >>         SharedCacheSettings.allowClear();
> >>
> >> AND
> >>
> >> public class SharedCacheStorage extends MruCacheStorage {
> >>     public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
> >>         super(maxStrongSize, maxSoftSize);
> >>     }
> >>
> >>     @Override
> >>     public void clear() {
> >>         if (!SharedCacheSettings.canClear())
> >>             return;
> >>
> >>         super.clear();
> >>     }
> >> }
> >>
> >> What is your opinion about my hack? What kind of trouble am I inviting?
> >>
> >> Thanks in advance.
> >> Jaime Garza
>

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: In 2.3.19, sharing cache storage

Daniel Dekany
Wednesday, July 23, 2014, 10:56:31 PM, Jaime Garza wrote:

>> -----Original Message-----
>> From: Daniel Dekany [mailto:[hidden email]]
>> Sent: Wednesday, July 23, 2014 1:06 PM
>> To: Jaime Garza
>> Cc: Daniel Dekany
>> Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>>
>> Wednesday, July 23, 2014, 8:48:28 PM, Jaime Garza wrote:
>>
>> > The modifications that I have done have to do with adaptations to FM
>> > to make it more of a DSL. We use FM for formatting emails and SMS
>> > messages. I have modified mostly FMParser.jj keeping a series of svn
>> > patch files for each one of my modifications, hopefully helping me
>> > move to a more recent FM. We have extensions for sql queries with a
>> > <#data> directive, data phrasing as a builtin for sequences, ?unique
>> > for sequences, etc.
>>
> Yeah I know that. We have a myriad of functions we implement. Our
> additions look, as you mention, terse. It appears as part of the
> language, which it now is. It would have been a dream to be able to
> add additional built-ins, and I have been following the list for a
> while now, and I know that this is a common ask and that you are
> thinking on ways to do it. I look forward to the moment we can have
> "legal" built-ins in the environment. Adding directives in a
> standard way with <# would have been ideal as well.

In the far future I plan to add namespace prefixes via colon. This is
very similar to #import at first glance, but has some important
benefits. One of them is that if you have a function (either defined
in FTL or in Java) then you can call it both like my:unique(seq) or
with postfix suntax, seq?my:unique. (Also then the "built-in" term is
gone - they are just core functions.) A side benefit is that
<#my:data/> perhaps looks less off than <@my.date/>. And of course,
just like #imports can be done on Configuration-level, prefixes could
be defined there too, so it doesn't have to be done in each template
again and again.

Now, if adding those "my:"-s (or whatever prefix you chose, obviously)
and maybe the explicit "/>"-s still seen as too awkward (is it?), this
idea could be further extended so that you can define your own
template language be directives/functions to the core namespace in the
Configuration level (read: pre-parse-time). I'm afraid of that though,
as then it's too easy to create custom languages, so people will do it
even when it's doesn't really worth the chaos that it causes later
heads (manual doesn't mach, copy-pasting from forums doesn't work, IDE
plugings complain, etc.). OTOH it would be useful when otherwise
someone really had to make a patched FreeMarker version.

>> You can have custom directives and functions without modifying FM, as you
>> certainly know. Surely they are not as flexible/terse as what can be achieved
>> with core directives and built-ins... Making them closer is a future goal.
>>
>> > We maintain our own FM document based on the
>> > 2.3.19 docs. We needed some validation support,
>>
>> Validation of what?
>
> Users declare their own namespaces in the UI. These are attached to
> database rows and appear in the root namespace. We need to know if
> they are really used in the templates at all, for backend
> optimizations.
> We warn users if they declare them and then they do not use them.

Sounds like that part can be solved by traversing the AST and collect
the variables referenced. (There's no publISHED API for that yet
though.)

> For this purpose we follow templates depth-first in a parsing phase
> as much as possible. If the include uses string expressions, we warn
> too, as we cannot proceed. We use the parse flag to follow includes.
> Etc. It is kind of elaborate, but it ended being very powerful.

This is maybe also solvable with AST traversal.

> In fact, I have been following the discussion of editor plugins and
> how we can get the parsing tree, and how people have done this in
> their own way, and then communicate with you. This is a way in which
> we discover the namespace for us, in-depth. I instrumented
> FMParser's PrimaryExpression and Include for this. I did not have to
> open DollarVariable.java or Include.java for it.
>
>>
>> > multiple error messages per template (one error was not acceptable,)
>>
>> Is that multiple parsing errors or?
>
> We do multiple parse errors, and lexical errors as well. We
> implemented a recover mechanism to stabilize the parsing process as
> much as possible, by skipping tokens, and backtracking. We then
> return a list of errors as opposed to one error. This was a big
> change.

This is also in the plans because of the IDE plugins.

So yeah... looks like all well known issues again. As always, the
problem is the lack of time.

--
Thanks,
 Daniel Dekany

>>
>> > and others.
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>> > Thanks
>> > Jaime
>> >
>> > -----Original Message-----
>> > From: Daniel Dekany [mailto:[hidden email]]
>> > Sent: Wednesday, July 23, 2014 11:22 AM
>> > To: Jaime Garza
>> > Cc: [hidden email]
>> > Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>> >
>> > The problem with the approach is that Template-s are bound to a
>> > Configuration instance (they store a reference to it), and sometimes
>> > may reach back to it during template execution. However, can't you
>> > just use a common Configuration instance on all those places that share
>> templates?
>> >
>> > (Out of curiosity... what's the modification in 2.3.19 is about?)
>> >
>> > --
>> > Thanks,
>> >  Daniel Dekany
>> >
>> > Wednesday, July 23, 2014, 5:51:52 PM, Jaime Garza wrote:
>> >
>> >> Hi Daniel,
>> >>
>> >> I am trying to share an instance of MruCacheStorage across threads.
>> >> I keep it around across multiple threads and when creating my
>> >> Configuration, I simply do a Configuration#setCacheStorage(). This
>> >> internally calls Configuration#createTemplateCache, which calls
>> >> cache.setConfiguration(this), which calls TemplateCache#clear(), which
>> >> eventually clears the storage.
>> >>
>> >> My use case is that I know that a set of around 200 threads will be
>> >> accessing the same templates over and over (executing the same
>> >> template(s) on a different root namespace) concurrently. I want the
>> >> loading of templates to happen as infrequently as possible. Also,
>> >> since I am on a slightly customized version of 2.3.19, I cannot come to the
>> greatest new version.
>> >>
>> >> The question (or questions) is: is there a way to implement my
>> >> strategy? My current workaround is nasty. I have created a subclass of
>> >> MruCacheStorage, which reads a flag "canClear" from thread local, and
>> >> then decides whether to let the clear proceed or not. I sandwitched
>> >> the call to setCacheStorage() with canClear = false and canClear =
>> >> true
>> >>
>> >>         SharedCacheSettings.disallowClear();
>> >>         config.setCacheStorage(storage);
>> >>         SharedCacheSettings.allowClear();
>> >>
>> >> AND
>> >>
>> >> public class SharedCacheStorage extends MruCacheStorage {
>> >>     public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
>> >>         super(maxStrongSize, maxSoftSize);
>> >>     }
>> >>
>> >>     @Override
>> >>     public void clear() {
>> >>         if (!SharedCacheSettings.canClear())
>> >>             return;
>> >>
>> >>         super.clear();
>> >>     }
>> >> }
>> >>
>> >> What is your opinion about my hack? What kind of trouble am I inviting?
>> >>
>> >> Thanks in advance.
>> >> Jaime Garza


------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: In 2.3.19, sharing cache storage

Jaime Garza-2
I think the my: prefix (or whatever I choose) would make it still "foreign" and not part of the language. My use case is that it is really a DSL that I need. In any case, I already have issues with copy/pasting from the site since ?join is there, and not in my 2.3.19 version. Others are coming that will not allow copy/paste for me.

But agreed. Having a DSL is an endeavor that requires tender love and care. We made a conscious decision to go this way, and certainly it is not for everyone. For us this DSL is core to our offering, so we can support it with a group of developers, writers, PMs, trainers, etc. Having said all that, most of my customization has been possible with new classes and only modifying two of your files: FMParser.jj, ParseException. However, I use package freemarker.core and make use extensively of those protected things you have there.

I understand that this is a matter of time from you. So far the results of your efforts are impressive though. Once I get a list of new goodies in a new version that I can sell to PMs, I will re-implement with that new version of FM.

My only ask would be if more classes were public and non-final. That would be lovely!

Jaime

-----Original Message-----
From: Daniel Dekany [mailto:[hidden email]]
Sent: Wednesday, July 23, 2014 3:20 PM
To: Jaime Garza
Cc: Daniel Dekany
Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage

Wednesday, July 23, 2014, 10:56:31 PM, Jaime Garza wrote:

>> -----Original Message-----
>> From: Daniel Dekany [mailto:[hidden email]]
>> Sent: Wednesday, July 23, 2014 1:06 PM
>> To: Jaime Garza
>> Cc: Daniel Dekany
>> Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>>
>> Wednesday, July 23, 2014, 8:48:28 PM, Jaime Garza wrote:
>>
>> > The modifications that I have done have to do with adaptations to
>> > FM to make it more of a DSL. We use FM for formatting emails and
>> > SMS messages. I have modified mostly FMParser.jj keeping a series
>> > of svn patch files for each one of my modifications, hopefully
>> > helping me move to a more recent FM. We have extensions for sql
>> > queries with a <#data> directive, data phrasing as a builtin for
>> > sequences, ?unique for sequences, etc.
>>
> Yeah I know that. We have a myriad of functions we implement. Our
> additions look, as you mention, terse. It appears as part of the
> language, which it now is. It would have been a dream to be able to
> add additional built-ins, and I have been following the list for a
> while now, and I know that this is a common ask and that you are
> thinking on ways to do it. I look forward to the moment we can have
> "legal" built-ins in the environment. Adding directives in a standard
> way with <# would have been ideal as well.

In the far future I plan to add namespace prefixes via colon. This is very similar to #import at first glance, but has some important benefits. One of them is that if you have a function (either defined in FTL or in Java) then you can call it both like my:unique(seq) or with postfix suntax, seq?my:unique. (Also then the "built-in" term is gone - they are just core functions.) A side benefit is that <#my:data/> perhaps looks less off than <@my.date/>. And of course, just like #imports can be done on Configuration-level, prefixes could be defined there too, so it doesn't have to be done in each template again and again.

Now, if adding those "my:"-s (or whatever prefix you chose, obviously) and maybe the explicit "/>"-s still seen as too awkward (is it?), this idea could be further extended so that you can define your own template language be directives/functions to the core namespace in the Configuration level (read: pre-parse-time). I'm afraid of that though, as then it's too easy to create custom languages, so people will do it even when it's doesn't really worth the chaos that it causes later heads (manual doesn't mach, copy-pasting from forums doesn't work, IDE plugings complain, etc.). OTOH it would be useful when otherwise someone really had to make a patched FreeMarker version.

>> You can have custom directives and functions without modifying FM, as
>> you certainly know. Surely they are not as flexible/terse as what can
>> be achieved with core directives and built-ins... Making them closer is a future goal.
>>
>> > We maintain our own FM document based on the
>> > 2.3.19 docs. We needed some validation support,
>>
>> Validation of what?
>
> Users declare their own namespaces in the UI. These are attached to
> database rows and appear in the root namespace. We need to know if
> they are really used in the templates at all, for backend
> optimizations.
> We warn users if they declare them and then they do not use them.

Sounds like that part can be solved by traversing the AST and collect the variables referenced. (There's no publISHED API for that yet
though.)

> For this purpose we follow templates depth-first in a parsing phase as
> much as possible. If the include uses string expressions, we warn too,
> as we cannot proceed. We use the parse flag to follow includes.
> Etc. It is kind of elaborate, but it ended being very powerful.

This is maybe also solvable with AST traversal.

> In fact, I have been following the discussion of editor plugins and
> how we can get the parsing tree, and how people have done this in
> their own way, and then communicate with you. This is a way in which
> we discover the namespace for us, in-depth. I instrumented FMParser's
> PrimaryExpression and Include for this. I did not have to open
> DollarVariable.java or Include.java for it.
>
>>
>> > multiple error messages per template (one error was not
>> > acceptable,)
>>
>> Is that multiple parsing errors or?
>
> We do multiple parse errors, and lexical errors as well. We
> implemented a recover mechanism to stabilize the parsing process as
> much as possible, by skipping tokens, and backtracking. We then return
> a list of errors as opposed to one error. This was a big change.

This is also in the plans because of the IDE plugins.

So yeah... looks like all well known issues again. As always, the problem is the lack of time.

--
Thanks,
 Daniel Dekany

>>
>> > and others.
>>
>> --
>> Thanks,
>>  Daniel Dekany
>>
>>
>> > Thanks
>> > Jaime
>> >
>> > -----Original Message-----
>> > From: Daniel Dekany [mailto:[hidden email]]
>> > Sent: Wednesday, July 23, 2014 11:22 AM
>> > To: Jaime Garza
>> > Cc: [hidden email]
>> > Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>> >
>> > The problem with the approach is that Template-s are bound to a
>> > Configuration instance (they store a reference to it), and
>> > sometimes may reach back to it during template execution. However,
>> > can't you just use a common Configuration instance on all those
>> > places that share
>> templates?
>> >
>> > (Out of curiosity... what's the modification in 2.3.19 is about?)
>> >
>> > --
>> > Thanks,
>> >  Daniel Dekany
>> >
>> > Wednesday, July 23, 2014, 5:51:52 PM, Jaime Garza wrote:
>> >
>> >> Hi Daniel,
>> >>
>> >> I am trying to share an instance of MruCacheStorage across threads.
>> >> I keep it around across multiple threads and when creating my
>> >> Configuration, I simply do a Configuration#setCacheStorage(). This
>> >> internally calls Configuration#createTemplateCache, which calls
>> >> cache.setConfiguration(this), which calls TemplateCache#clear(),
>> >> which eventually clears the storage.
>> >>
>> >> My use case is that I know that a set of around 200 threads will
>> >> be accessing the same templates over and over (executing the same
>> >> template(s) on a different root namespace) concurrently. I want
>> >> the loading of templates to happen as infrequently as possible.
>> >> Also, since I am on a slightly customized version of 2.3.19, I
>> >> cannot come to the
>> greatest new version.
>> >>
>> >> The question (or questions) is: is there a way to implement my
>> >> strategy? My current workaround is nasty. I have created a
>> >> subclass of MruCacheStorage, which reads a flag "canClear" from
>> >> thread local, and then decides whether to let the clear proceed or
>> >> not. I sandwitched the call to setCacheStorage() with canClear =
>> >> false and canClear = true
>> >>
>> >>         SharedCacheSettings.disallowClear();
>> >>         config.setCacheStorage(storage);
>> >>         SharedCacheSettings.allowClear();
>> >>
>> >> AND
>> >>
>> >> public class SharedCacheStorage extends MruCacheStorage {
>> >>     public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
>> >>         super(maxStrongSize, maxSoftSize);
>> >>     }
>> >>
>> >>     @Override
>> >>     public void clear() {
>> >>         if (!SharedCacheSettings.canClear())
>> >>             return;
>> >>
>> >>         super.clear();
>> >>     }
>> >> }
>> >>
>> >> What is your opinion about my hack? What kind of trouble am I inviting?
>> >>
>> >> Thanks in advance.
>> >> Jaime Garza


------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: In 2.3.19, sharing cache storage

Daniel Dekany
Thursday, July 24, 2014, 12:40:21 AM, Jaime Garza wrote:

> I think the my: prefix (or whatever I choose) would make it still
> "foreign" and not part of the language. My use case is that it is
> really a DSL that I need. In any case, I already have issues with
> copy/pasting from the site since ?join is there, and not in my
> 2.3.19 version. Others are coming that will not allow copy/paste for me.
>
> But agreed. Having a DSL is an endeavor that requires tender love
> and care. We made a conscious decision to go this way, and certainly
> it is not for everyone. For us this DSL is core to our offering, so
> we can support it with a group of developers, writers, PMs,
> trainers, etc. Having said all that, most of my customization has
> been possible with new classes and only modifying two of your files:
> FMParser.jj, ParseException. However, I use package freemarker.core
> and make use extensively of those protected things you have there.
>
> I understand that this is a matter of time from you.

(I "only" had to work on FM full time instead of a few hours a week...
It's funny how much is there to do in a small template engine.)

> So far the results of your efforts are impressive though. Once I get
> a list of new goodies in a new version that I can sell to PMs, I
> will re-implement with that new version of FM.
>
> My only ask would be if more classes were public and non-final. That would be lovely!

The more I expose from the internal(ish...) classes, the more
difficult will it be to fix/evolve anything without breaking backward
compatibility. It become obvious that traversing the AST would be
important for many users. Even modifying it for some. However,
exposing the AST directly would be a rather risky step. Instead, FM
will have to introduce another library, like, freemarker-inspector or
something, that defines a logical AST-like view of a template with a
published API, and which depends on the FreeMarker internals, and is
maintained together with FreeMarker... Or what other API would you
need to be published?

> Jaime
>
> -----Original Message-----
> From: Daniel Dekany [mailto:[hidden email]]
> Sent: Wednesday, July 23, 2014 3:20 PM
> To: Jaime Garza
> Cc: Daniel Dekany
> Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>
> Wednesday, July 23, 2014, 10:56:31 PM, Jaime Garza wrote:
>
>>> -----Original Message-----
>>> From: Daniel Dekany [mailto:[hidden email]]
>>> Sent: Wednesday, July 23, 2014 1:06 PM
>>> To: Jaime Garza
>>> Cc: Daniel Dekany
>>> Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>>>
>>> Wednesday, July 23, 2014, 8:48:28 PM, Jaime Garza wrote:
>>>
>>> > The modifications that I have done have to do with adaptations to
>>> > FM to make it more of a DSL. We use FM for formatting emails and
>>> > SMS messages. I have modified mostly FMParser.jj keeping a series
>>> > of svn patch files for each one of my modifications, hopefully
>>> > helping me move to a more recent FM. We have extensions for sql
>>> > queries with a <#data> directive, data phrasing as a builtin for
>>> > sequences, ?unique for sequences, etc.
>>>
>> Yeah I know that. We have a myriad of functions we implement. Our
>> additions look, as you mention, terse. It appears as part of the
>> language, which it now is. It would have been a dream to be able to
>> add additional built-ins, and I have been following the list for a
>> while now, and I know that this is a common ask and that you are
>> thinking on ways to do it. I look forward to the moment we can have
>> "legal" built-ins in the environment. Adding directives in a standard
>> way with <# would have been ideal as well.
>
> In the far future I plan to add namespace prefixes via colon. This
> is very similar to #import at first glance, but has some important
> benefits. One of them is that if you have a function (either defined
> in FTL or in Java) then you can call it both like my:unique(seq) or
> with postfix suntax, seq?my:unique. (Also then the "built-in" term
> is gone - they are just core functions.) A side benefit is that
> <#my:data/> perhaps looks less off than <@my.date/>. And of course,
> just like #imports can be done on Configuration-level, prefixes
> could be defined there too, so it doesn't have to be done in each template again and again.
>
> Now, if adding those "my:"-s (or whatever prefix you chose,
> obviously) and maybe the explicit "/>"-s still seen as too awkward
> (is it?), this idea could be further extended so that you can define
> your own template language be directives/functions to the core
> namespace in the Configuration level (read: pre-parse-time). I'm
> afraid of that though, as then it's too easy to create custom
> languages, so people will do it even when it's doesn't really worth
> the chaos that it causes later heads (manual doesn't mach,
> copy-pasting from forums doesn't work, IDE plugings complain, etc.).
> OTOH it would be useful when otherwise someone really had to make a patched FreeMarker version.
>
>>> You can have custom directives and functions without modifying FM, as
>>> you certainly know. Surely they are not as flexible/terse as what can
>>> be achieved with core directives and built-ins... Making them closer is a future goal.
>>>
>>> > We maintain our own FM document based on the
>>> > 2.3.19 docs. We needed some validation support,
>>>
>>> Validation of what?
>>
>> Users declare their own namespaces in the UI. These are attached to
>> database rows and appear in the root namespace. We need to know if
>> they are really used in the templates at all, for backend
>> optimizations.
>> We warn users if they declare them and then they do not use them.
>
> Sounds like that part can be solved by traversing the AST and
> collect the variables referenced. (There's no publISHED API for that yet
> though.)
>
>> For this purpose we follow templates depth-first in a parsing phase as
>> much as possible. If the include uses string expressions, we warn too,
>> as we cannot proceed. We use the parse flag to follow includes.
>> Etc. It is kind of elaborate, but it ended being very powerful.
>
> This is maybe also solvable with AST traversal.
>
>> In fact, I have been following the discussion of editor plugins and
>> how we can get the parsing tree, and how people have done this in
>> their own way, and then communicate with you. This is a way in which
>> we discover the namespace for us, in-depth. I instrumented FMParser's
>> PrimaryExpression and Include for this. I did not have to open
>> DollarVariable.java or Include.java for it.
>>
>>>
>>> > multiple error messages per template (one error was not
>>> > acceptable,)
>>>
>>> Is that multiple parsing errors or?
>>
>> We do multiple parse errors, and lexical errors as well. We
>> implemented a recover mechanism to stabilize the parsing process as
>> much as possible, by skipping tokens, and backtracking. We then return
>> a list of errors as opposed to one error. This was a big change.
>
> This is also in the plans because of the IDE plugins.
>
> So yeah... looks like all well known issues again. As always, the problem is the lack of time.
>
> --
> Thanks,
>  Daniel Dekany
>
>>>
>>> > and others.
>>>
>>> --
>>> Thanks,
>>>  Daniel Dekany
>>>
>>>
>>> > Thanks
>>> > Jaime
>>> >
>>> > -----Original Message-----
>>> > From: Daniel Dekany [mailto:[hidden email]]
>>> > Sent: Wednesday, July 23, 2014 11:22 AM
>>> > To: Jaime Garza
>>> > Cc: [hidden email]
>>> > Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage
>>> >
>>> > The problem with the approach is that Template-s are bound to a
>>> > Configuration instance (they store a reference to it), and
>>> > sometimes may reach back to it during template execution. However,
>>> > can't you just use a common Configuration instance on all those
>>> > places that share
>>> templates?
>>> >
>>> > (Out of curiosity... what's the modification in 2.3.19 is about?)
>>> >
>>> > --
>>> > Thanks,
>>> >  Daniel Dekany
>>> >
>>> > Wednesday, July 23, 2014, 5:51:52 PM, Jaime Garza wrote:
>>> >
>>> >> Hi Daniel,
>>> >>
>>> >> I am trying to share an instance of MruCacheStorage across threads.
>>> >> I keep it around across multiple threads and when creating my
>>> >> Configuration, I simply do a Configuration#setCacheStorage(). This
>>> >> internally calls Configuration#createTemplateCache, which calls
>>> >> cache.setConfiguration(this), which calls TemplateCache#clear(),
>>> >> which eventually clears the storage.
>>> >>
>>> >> My use case is that I know that a set of around 200 threads will
>>> >> be accessing the same templates over and over (executing the same
>>> >> template(s) on a different root namespace) concurrently. I want
>>> >> the loading of templates to happen as infrequently as possible.
>>> >> Also, since I am on a slightly customized version of 2.3.19, I
>>> >> cannot come to the
>>> greatest new version.
>>> >>
>>> >> The question (or questions) is: is there a way to implement my
>>> >> strategy? My current workaround is nasty. I have created a
>>> >> subclass of MruCacheStorage, which reads a flag "canClear" from
>>> >> thread local, and then decides whether to let the clear proceed or
>>> >> not. I sandwitched the call to setCacheStorage() with canClear =
>>> >> false and canClear = true
>>> >>
>>> >>         SharedCacheSettings.disallowClear();
>>> >>         config.setCacheStorage(storage);
>>> >>         SharedCacheSettings.allowClear();
>>> >>
>>> >> AND
>>> >>
>>> >> public class SharedCacheStorage extends MruCacheStorage {
>>> >>     public SharedCacheStorage(int maxStrongSize, int maxSoftSize) {
>>> >>         super(maxStrongSize, maxSoftSize);
>>> >>     }
>>> >>
>>> >>     @Override
>>> >>     public void clear() {
>>> >>         if (!SharedCacheSettings.canClear())
>>> >>             return;
>>> >>
>>> >>         super.clear();
>>> >>     }
>>> >> }
>>> >>
>>> >> What is your opinion about my hack? What kind of trouble am I inviting?
>>> >>
>>> >> Thanks in advance.
>>> >> Jaime Garza
>
>
> ------------------------------------------------------------------------------
> Want fast and easy access to all the code in your enterprise? Index and
> search up to 200,000 lines of code with a free copy of Black Duck
> Code Sight - the same software that powers the world's largest code
> search on Ohloh, the Black Duck Open Hub! Try it now.
> http://p.sf.net/sfu/bds
> _______________________________________________
> FreeMarker-devel mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/freemarker-devel
>

--
Thanks,
 Daniel Dekany


------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: In 2.3.19, sharing cache storage

Wong, Christopher
If being able to inspect the AST was the only objective, I think my current freemarker-introspection projection would be of interest to Jaime:

        https://github.com/cwong15/freemarker-introspection

But my impression is that Jaime actually wants to change the behavior of Freemarker itself for his needs. In which case, what he wants is the ability to extend Freemarker, which is a whole different set of needs from simply being able to passively inspect a template outside of Freemarker's template processing.

Chris

-----Original Message-----
From: Daniel Dekany [mailto:[hidden email]]
Sent: Thursday, July 24, 2014 8:32 PM
To: Jaime Garza
Cc: Daniel Dekany
Subject: Re: [Freemarker-devel] In 2.3.19, sharing cache storage

Thursday, July 24, 2014, 12:40:21 AM, Jaime Garza wrote:
> My only ask would be if more classes were public and non-final. That would be lovely!

The more I expose from the internal(ish...) classes, the more difficult will it be to fix/evolve anything without breaking backward compatibility. It become obvious that traversing the AST would be important for many users. Even modifying it for some. However, exposing the AST directly would be a rather risky step. Instead, FM will have to introduce another library, like, freemarker-inspector or something, that defines a logical AST-like view of a template with a published API, and which depends on the FreeMarker internals, and is maintained together with FreeMarker... Or what other API would you need to be published?


This e-mail and files transmitted with it are confidential, and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you are not one of the named recipient(s) or otherwise have reason to believe that you received this message in error, please immediately notify sender by e-mail, and destroy the original message. Thank You.

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
FreeMarker-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/freemarker-devel
Loading...